Jul 28 01:15:42.452336 ip-10-0-131-23 systemd[1]: Starting Kubernetes Kubelet... Jul 28 01:15:42.840319 ip-10-0-131-23 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 01:15:42.840319 ip-10-0-131-23 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 28 01:15:42.840319 ip-10-0-131-23 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 01:15:42.840319 ip-10-0-131-23 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 28 01:15:42.840319 ip-10-0-131-23 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 01:15:42.841997 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.841872 2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 28 01:15:42.848630 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848608 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 01:15:42.848630 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848626 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 01:15:42.848630 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848631 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 01:15:42.848630 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848634 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 01:15:42.848630 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848637 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 01:15:42.848831 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848641 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 01:15:42.848831 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848644 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 01:15:42.848831 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848649 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 01:15:42.848831 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848651 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 01:15:42.848831 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848679 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 01:15:42.848831 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848683 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 01:15:42.848831 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848686 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 01:15:42.848831 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848688 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 01:15:42.848831 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848691 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 01:15:42.848831 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848694 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 01:15:42.848831 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848697 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 01:15:42.848831 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848699 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 01:15:42.848831 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848702 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 01:15:42.848831 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848705 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 01:15:42.848831 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848707 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 01:15:42.848831 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848710 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 01:15:42.848831 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848712 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 01:15:42.848831 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848715 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 01:15:42.848831 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848718 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 01:15:42.848831 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848721 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 01:15:42.849324 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848724 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 01:15:42.849324 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848726 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 01:15:42.849324 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848730 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 01:15:42.849324 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848732 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 01:15:42.849324 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848735 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 01:15:42.849324 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848738 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 01:15:42.849324 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848740 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 01:15:42.849324 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848744 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 01:15:42.849324 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848746 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 01:15:42.849324 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848749 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 01:15:42.849324 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848751 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 01:15:42.849324 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848754 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 01:15:42.849324 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848757 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 01:15:42.849324 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848760 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 01:15:42.849324 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848763 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 01:15:42.849324 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848765 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 01:15:42.849324 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848768 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 01:15:42.849324 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848771 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 01:15:42.849324 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848773 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 01:15:42.849324 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848776 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 01:15:42.849815 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848778 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 28 01:15:42.849815 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848781 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 01:15:42.849815 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848784 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 01:15:42.849815 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848786 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 01:15:42.849815 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848789 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 01:15:42.849815 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848791 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 01:15:42.849815 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848794 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 01:15:42.849815 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848797 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 01:15:42.849815 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848800 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 01:15:42.849815 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848802 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 01:15:42.849815 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848805 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 01:15:42.849815 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848808 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 01:15:42.849815 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848813 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 01:15:42.849815 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848816 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 01:15:42.849815 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848819 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 01:15:42.849815 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848822 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 01:15:42.849815 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848825 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 01:15:42.849815 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848828 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 01:15:42.849815 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848832 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 01:15:42.850380 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848834 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 01:15:42.850380 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848837 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 01:15:42.850380 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848840 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 01:15:42.850380 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848843 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 01:15:42.850380 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848846 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 01:15:42.850380 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848849 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 01:15:42.850380 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848851 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 01:15:42.850380 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848854 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 01:15:42.850380 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848856 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 01:15:42.850380 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848859 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 01:15:42.850380 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848862 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 01:15:42.850380 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848864 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 01:15:42.850380 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848867 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 01:15:42.850380 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848870 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 01:15:42.850380 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848872 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 01:15:42.850380 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848875 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 01:15:42.850380 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848877 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 01:15:42.850380 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848880 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 01:15:42.850380 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848882 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 01:15:42.850380 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848885 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 01:15:42.850885 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848888 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 01:15:42.850885 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848891 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 01:15:42.850885 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848893 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 01:15:42.850885 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848896 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 01:15:42.850885 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848898 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 01:15:42.850885 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848903 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 01:15:42.850885 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848907 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 01:15:42.850885 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848910 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 01:15:42.850885 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848914 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 01:15:42.850885 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848918 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 01:15:42.850885 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848922 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 01:15:42.850885 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848925 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 01:15:42.850885 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848927 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 01:15:42.850885 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848930 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 01:15:42.850885 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.848933 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 01:15:42.850885 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849671 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 01:15:42.850885 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849678 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 01:15:42.850885 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849681 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 01:15:42.850885 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849684 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 01:15:42.851428 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849687 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 01:15:42.851428 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849690 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 01:15:42.851428 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849692 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 01:15:42.851428 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849695 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 01:15:42.851428 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849698 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 01:15:42.851428 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849700 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 01:15:42.851428 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849705 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 01:15:42.851428 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849708 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 01:15:42.851428 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849712 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 01:15:42.851428 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849715 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 01:15:42.851428 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849718 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 01:15:42.851428 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849720 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 01:15:42.851428 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849723 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 01:15:42.851428 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849729 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 01:15:42.851428 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849732 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 01:15:42.851428 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849734 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 01:15:42.851428 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849738 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 01:15:42.851428 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849740 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 01:15:42.851428 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849743 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 01:15:42.851428 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849746 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 01:15:42.851921 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849749 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 01:15:42.851921 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849751 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 01:15:42.851921 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849754 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 01:15:42.851921 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849757 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 01:15:42.851921 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849759 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 01:15:42.851921 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849762 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 01:15:42.851921 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849764 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 01:15:42.851921 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849767 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 01:15:42.851921 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849769 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 01:15:42.851921 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849772 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 01:15:42.851921 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849775 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 01:15:42.851921 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849778 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 01:15:42.851921 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849780 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 01:15:42.851921 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849783 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 01:15:42.851921 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849786 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 01:15:42.851921 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849788 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 01:15:42.851921 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849791 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 01:15:42.851921 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849793 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 01:15:42.851921 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849796 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 01:15:42.851921 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849799 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 01:15:42.852445 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849801 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 01:15:42.852445 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849804 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 01:15:42.852445 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849806 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 01:15:42.852445 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849809 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 01:15:42.852445 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849812 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 28 01:15:42.852445 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849814 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 01:15:42.852445 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849817 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 01:15:42.852445 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849820 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 01:15:42.852445 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849825 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 01:15:42.852445 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849828 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 01:15:42.852445 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849831 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 01:15:42.852445 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849834 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 01:15:42.852445 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849836 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 01:15:42.852445 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849839 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 01:15:42.852445 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849842 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 01:15:42.852445 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849844 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 01:15:42.852445 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849847 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 01:15:42.852445 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849849 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 01:15:42.852445 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849852 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 01:15:42.852445 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849855 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 01:15:42.852941 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849857 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 01:15:42.852941 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849860 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 01:15:42.852941 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849864 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 01:15:42.852941 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849868 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 01:15:42.852941 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849872 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 01:15:42.852941 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849874 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 01:15:42.852941 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849877 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 01:15:42.852941 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849880 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 01:15:42.852941 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849883 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 01:15:42.852941 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849885 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 01:15:42.852941 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849888 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 01:15:42.852941 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849890 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 01:15:42.852941 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849893 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 01:15:42.852941 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849896 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 01:15:42.852941 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849899 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 01:15:42.852941 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849901 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 01:15:42.852941 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849904 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 01:15:42.852941 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849906 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 01:15:42.852941 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849909 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 01:15:42.853426 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849913 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 01:15:42.853426 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849916 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 01:15:42.853426 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849918 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 01:15:42.853426 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849921 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 01:15:42.853426 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849924 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 01:15:42.853426 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849926 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 01:15:42.853426 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849929 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 01:15:42.853426 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849931 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 01:15:42.853426 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849935 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 01:15:42.853426 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849938 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 01:15:42.853426 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849941 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 01:15:42.853426 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849944 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 01:15:42.853426 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849947 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 01:15:42.853426 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849950 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 01:15:42.853426 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849953 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 01:15:42.853426 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.849956 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 01:15:42.853426 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850357 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 01:15:42.853426 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850364 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 01:15:42.853426 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850367 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 01:15:42.853887 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850369 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 01:15:42.853887 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850373 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 01:15:42.853887 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850375 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 01:15:42.853887 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850378 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 01:15:42.853887 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850381 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 01:15:42.853887 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850383 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 01:15:42.853887 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850386 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 01:15:42.853887 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850390 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 01:15:42.853887 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850394 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 01:15:42.853887 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850397 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 01:15:42.853887 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850400 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 01:15:42.853887 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850403 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 01:15:42.853887 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850405 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 01:15:42.853887 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850408 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 01:15:42.853887 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850411 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 01:15:42.853887 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850414 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 01:15:42.853887 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850416 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 01:15:42.853887 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850419 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 01:15:42.853887 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850421 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 01:15:42.854359 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850425 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 01:15:42.854359 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850429 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 01:15:42.854359 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850432 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 01:15:42.854359 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850435 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 01:15:42.854359 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850438 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 01:15:42.854359 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850441 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 01:15:42.854359 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850444 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 01:15:42.854359 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850447 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 01:15:42.854359 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850450 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 01:15:42.854359 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850453 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 01:15:42.854359 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850455 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 01:15:42.854359 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850458 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 01:15:42.854359 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850460 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 01:15:42.854359 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850463 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 01:15:42.854359 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850465 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 01:15:42.854359 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850468 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 01:15:42.854359 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850470 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 01:15:42.854359 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850473 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 01:15:42.854359 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850475 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 01:15:42.854813 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850478 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 01:15:42.854813 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850481 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 01:15:42.854813 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850483 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 01:15:42.854813 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850485 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 01:15:42.854813 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850488 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 01:15:42.854813 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850491 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 01:15:42.854813 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850493 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 01:15:42.854813 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850496 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 01:15:42.854813 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850500 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 01:15:42.854813 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850503 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 01:15:42.854813 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850505 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 01:15:42.854813 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850508 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 01:15:42.854813 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850510 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 01:15:42.854813 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850512 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 01:15:42.854813 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850515 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 01:15:42.854813 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850518 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 28 01:15:42.854813 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850520 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 01:15:42.854813 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850523 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 01:15:42.854813 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850528 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 01:15:42.854813 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850531 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 01:15:42.855316 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850534 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 01:15:42.855316 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850536 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 01:15:42.855316 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850539 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 01:15:42.855316 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850541 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 01:15:42.855316 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850544 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 01:15:42.855316 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850548 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 01:15:42.855316 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850551 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 01:15:42.855316 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850554 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 01:15:42.855316 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850556 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 01:15:42.855316 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850559 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 01:15:42.855316 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850563 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 01:15:42.855316 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850566 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 01:15:42.855316 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850568 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 01:15:42.855316 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850571 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 01:15:42.855316 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850574 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 01:15:42.855316 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850576 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 01:15:42.855316 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850579 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 01:15:42.855316 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850581 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 01:15:42.855316 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850584 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 01:15:42.855803 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850586 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 01:15:42.855803 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850589 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 01:15:42.855803 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850592 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 01:15:42.855803 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850595 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 01:15:42.855803 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850597 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 01:15:42.855803 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850600 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 01:15:42.855803 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850603 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 01:15:42.855803 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850605 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 01:15:42.855803 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850608 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 01:15:42.855803 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850610 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 01:15:42.855803 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850613 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 01:15:42.855803 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850617 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 01:15:42.855803 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850620 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 01:15:42.855803 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850622 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 01:15:42.855803 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850625 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 01:15:42.855803 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850628 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 01:15:42.855803 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850630 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 01:15:42.855803 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850632 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 01:15:42.855803 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.850635 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 01:15:42.855803 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850698 2565 flags.go:64] FLAG: --address="0.0.0.0" Jul 28 01:15:42.855803 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850709 2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 28 01:15:42.856332 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850714 2565 flags.go:64] FLAG: --anonymous-auth="true" Jul 28 01:15:42.856332 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850719 2565 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 28 01:15:42.856332 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850723 2565 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 28 01:15:42.856332 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850726 2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 28 01:15:42.856332 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850730 2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 28 01:15:42.856332 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850734 2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 28 01:15:42.856332 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850737 2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 28 01:15:42.856332 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850740 2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 28 01:15:42.856332 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850744 2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 28 01:15:42.856332 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850747 2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 28 01:15:42.856332 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850751 2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 28 01:15:42.856332 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850754 2565 flags.go:64] FLAG: --cgroup-root="" Jul 28 01:15:42.856332 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850757 2565 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 28 01:15:42.856332 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850760 2565 flags.go:64] FLAG: --client-ca-file="" Jul 28 01:15:42.856332 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850764 2565 flags.go:64] FLAG: --cloud-provider="external" Jul 28 01:15:42.856332 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850767 2565 flags.go:64] FLAG: --cluster-dns="[]" Jul 28 01:15:42.856332 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850771 2565 flags.go:64] FLAG: --cluster-domain="" Jul 28 01:15:42.856332 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850774 2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 28 01:15:42.856332 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850778 2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 28 01:15:42.856332 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850781 2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 28 01:15:42.856332 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850784 2565 flags.go:64] FLAG: --container-log-max-files="5" Jul 28 01:15:42.856332 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850788 2565 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 28 01:15:42.856332 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850792 2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 28 01:15:42.856880 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850795 2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 28 01:15:42.856880 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850798 2565 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 28 01:15:42.856880 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850801 2565 flags.go:64] FLAG: --contention-profiling="false" Jul 28 01:15:42.856880 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850804 2565 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 28 01:15:42.856880 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850807 2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 28 01:15:42.856880 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850810 2565 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 28 01:15:42.856880 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850813 2565 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 28 01:15:42.856880 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850817 2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 28 01:15:42.856880 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850820 2565 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 28 01:15:42.856880 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850823 2565 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 28 01:15:42.856880 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850826 2565 flags.go:64] FLAG: --enable-load-reader="false" Jul 28 01:15:42.856880 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850829 2565 flags.go:64] FLAG: --enable-server="true" Jul 28 01:15:42.856880 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850832 2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 28 01:15:42.856880 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850836 2565 flags.go:64] FLAG: --event-burst="100" Jul 28 01:15:42.856880 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850841 2565 flags.go:64] FLAG: --event-qps="50" Jul 28 01:15:42.856880 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850846 2565 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 28 01:15:42.856880 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850850 2565 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 28 01:15:42.856880 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850855 2565 flags.go:64] FLAG: --eviction-hard="" Jul 28 01:15:42.856880 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850860 2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 28 01:15:42.856880 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850865 2565 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 28 01:15:42.856880 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850870 2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 28 01:15:42.856880 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850875 2565 flags.go:64] FLAG: --eviction-soft="" Jul 28 01:15:42.856880 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850878 2565 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 28 01:15:42.856880 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850881 2565 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 28 01:15:42.856880 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850885 2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 28 01:15:42.857505 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850888 2565 flags.go:64] FLAG: --experimental-mounter-path="" Jul 28 01:15:42.857505 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850891 2565 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 28 01:15:42.857505 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850894 2565 flags.go:64] FLAG: --fail-swap-on="true" Jul 28 01:15:42.857505 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850897 2565 flags.go:64] FLAG: --feature-gates="" Jul 28 01:15:42.857505 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850903 2565 flags.go:64] FLAG: --file-check-frequency="20s" Jul 28 01:15:42.857505 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850906 2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 28 01:15:42.857505 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850910 2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 28 01:15:42.857505 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850914 2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 28 01:15:42.857505 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850917 2565 flags.go:64] FLAG: --healthz-port="10248" Jul 28 01:15:42.857505 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850920 2565 flags.go:64] FLAG: --help="false" Jul 28 01:15:42.857505 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850923 2565 flags.go:64] FLAG: --hostname-override="ip-10-0-131-23.ec2.internal" Jul 28 01:15:42.857505 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850926 2565 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 28 01:15:42.857505 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850929 2565 flags.go:64] FLAG: --http-check-frequency="20s" Jul 28 01:15:42.857505 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850932 2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 28 01:15:42.857505 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850935 2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 28 01:15:42.857505 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850938 2565 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 28 01:15:42.857505 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850941 2565 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 28 01:15:42.857505 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850944 2565 flags.go:64] FLAG: --image-service-endpoint="" Jul 28 01:15:42.857505 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850947 2565 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 28 01:15:42.857505 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850950 2565 flags.go:64] FLAG: --kube-api-burst="100" Jul 28 01:15:42.857505 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850953 2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 28 01:15:42.857505 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850956 2565 flags.go:64] FLAG: --kube-api-qps="50" Jul 28 01:15:42.857505 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850959 2565 flags.go:64] FLAG: --kube-reserved="" Jul 28 01:15:42.857505 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850961 2565 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 28 01:15:42.858096 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850964 2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 28 01:15:42.858096 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850967 2565 flags.go:64] FLAG: --kubelet-cgroups="" Jul 28 01:15:42.858096 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850970 2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 28 01:15:42.858096 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850973 2565 flags.go:64] FLAG: --lock-file="" Jul 28 01:15:42.858096 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850976 2565 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 28 01:15:42.858096 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850979 2565 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 28 01:15:42.858096 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850982 2565 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 28 01:15:42.858096 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850987 2565 flags.go:64] FLAG: --log-json-split-stream="false" Jul 28 01:15:42.858096 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850990 2565 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 28 01:15:42.858096 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850993 2565 flags.go:64] FLAG: --log-text-split-stream="false" Jul 28 01:15:42.858096 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850996 2565 flags.go:64] FLAG: --logging-format="text" Jul 28 01:15:42.858096 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.850999 2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 28 01:15:42.858096 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851004 2565 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 28 01:15:42.858096 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851007 2565 flags.go:64] FLAG: --manifest-url="" Jul 28 01:15:42.858096 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851011 2565 flags.go:64] FLAG: --manifest-url-header="" Jul 28 01:15:42.858096 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851015 2565 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 28 01:15:42.858096 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851018 2565 flags.go:64] FLAG: --max-open-files="1000000" Jul 28 01:15:42.858096 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851022 2565 flags.go:64] FLAG: --max-pods="110" Jul 28 01:15:42.858096 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851025 2565 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 28 01:15:42.858096 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851028 2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 28 01:15:42.858096 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851044 2565 flags.go:64] FLAG: --memory-manager-policy="None" Jul 28 01:15:42.858096 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851047 2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 28 01:15:42.858096 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851050 2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 28 01:15:42.858096 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851053 2565 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 28 01:15:42.858096 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851056 2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 28 01:15:42.858711 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851063 2565 flags.go:64] FLAG: --node-status-max-images="50" Jul 28 01:15:42.858711 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851066 2565 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 28 01:15:42.858711 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851069 2565 flags.go:64] FLAG: --oom-score-adj="-999" Jul 28 01:15:42.858711 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851072 2565 flags.go:64] FLAG: --pod-cidr="" Jul 28 01:15:42.858711 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851075 2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 28 01:15:42.858711 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851080 2565 flags.go:64] FLAG: --pod-manifest-path="" Jul 28 01:15:42.858711 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851083 2565 flags.go:64] FLAG: --pod-max-pids="-1" Jul 28 01:15:42.858711 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851086 2565 flags.go:64] FLAG: --pods-per-core="0" Jul 28 01:15:42.858711 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851089 2565 flags.go:64] FLAG: --port="10250" Jul 28 01:15:42.858711 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851092 2565 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 28 01:15:42.858711 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851095 2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0630841f489cc3bc8" Jul 28 01:15:42.858711 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851098 2565 flags.go:64] FLAG: --qos-reserved="" Jul 28 01:15:42.858711 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851101 2565 flags.go:64] FLAG: --read-only-port="10255" Jul 28 01:15:42.858711 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851104 2565 flags.go:64] FLAG: --register-node="true" Jul 28 01:15:42.858711 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851107 2565 flags.go:64] FLAG: --register-with-taints="" Jul 28 01:15:42.858711 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851111 2565 flags.go:64] FLAG: --registry-burst="10" Jul 28 01:15:42.858711 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851115 2565 flags.go:64] FLAG: --registry-qps="5" Jul 28 01:15:42.858711 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851118 2565 flags.go:64] FLAG: --reserved-cpus="" Jul 28 01:15:42.858711 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851121 2565 flags.go:64] FLAG: --reserved-memory="" Jul 28 01:15:42.858711 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851126 2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 28 01:15:42.858711 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851129 2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 28 01:15:42.858711 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851132 2565 flags.go:64] FLAG: --rotate-certificates="false" Jul 28 01:15:42.858711 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851137 2565 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 28 01:15:42.858711 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851140 2565 flags.go:64] FLAG: --runonce="false" Jul 28 01:15:42.858711 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851142 2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 28 01:15:42.859355 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851145 2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 28 01:15:42.859355 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851148 2565 flags.go:64] FLAG: --seccomp-default="false" Jul 28 01:15:42.859355 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851151 2565 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 28 01:15:42.859355 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851154 2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 28 01:15:42.859355 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851157 2565 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 28 01:15:42.859355 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851160 2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 28 01:15:42.859355 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851163 2565 flags.go:64] FLAG: --storage-driver-password="root" Jul 28 01:15:42.859355 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851166 2565 flags.go:64] FLAG: --storage-driver-secure="false" Jul 28 01:15:42.859355 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851169 2565 flags.go:64] FLAG: --storage-driver-table="stats" Jul 28 01:15:42.859355 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851172 2565 flags.go:64] FLAG: --storage-driver-user="root" Jul 28 01:15:42.859355 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851174 2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 28 01:15:42.859355 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851177 2565 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 28 01:15:42.859355 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851180 2565 flags.go:64] FLAG: --system-cgroups="" Jul 28 01:15:42.859355 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851183 2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 28 01:15:42.859355 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851188 2565 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 28 01:15:42.859355 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851191 2565 flags.go:64] FLAG: --tls-cert-file="" Jul 28 01:15:42.859355 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851194 2565 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 28 01:15:42.859355 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851198 2565 flags.go:64] FLAG: --tls-min-version="" Jul 28 01:15:42.859355 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851201 2565 flags.go:64] FLAG: --tls-private-key-file="" Jul 28 01:15:42.859355 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851204 2565 flags.go:64] FLAG: --topology-manager-policy="none" Jul 28 01:15:42.859355 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851207 2565 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 28 01:15:42.859355 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851210 2565 flags.go:64] FLAG: --topology-manager-scope="container" Jul 28 01:15:42.859355 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851212 2565 flags.go:64] FLAG: --v="2" Jul 28 01:15:42.859355 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851216 2565 flags.go:64] FLAG: --version="false" Jul 28 01:15:42.859355 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851220 2565 flags.go:64] FLAG: --vmodule="" Jul 28 01:15:42.859952 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851224 2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 28 01:15:42.859952 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.851229 2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 28 01:15:42.859952 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851359 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 01:15:42.859952 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851363 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 01:15:42.859952 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851368 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 28 01:15:42.859952 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851370 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 01:15:42.859952 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851374 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 01:15:42.859952 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851378 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 01:15:42.859952 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851382 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 01:15:42.859952 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851385 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 01:15:42.859952 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851388 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 01:15:42.859952 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851391 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 01:15:42.859952 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851394 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 01:15:42.859952 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851397 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 01:15:42.859952 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851400 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 01:15:42.859952 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851403 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 01:15:42.859952 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851405 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 01:15:42.859952 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851408 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 01:15:42.859952 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851410 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 01:15:42.859952 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851413 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 01:15:42.860475 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851415 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 01:15:42.860475 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851418 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 01:15:42.860475 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851421 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 01:15:42.860475 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851423 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 01:15:42.860475 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851426 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 01:15:42.860475 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851428 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 01:15:42.860475 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851431 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 01:15:42.860475 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851433 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 01:15:42.860475 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851436 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 01:15:42.860475 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851439 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 01:15:42.860475 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851442 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 01:15:42.860475 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851444 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 01:15:42.860475 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851448 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 01:15:42.860475 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851451 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 01:15:42.860475 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851454 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 01:15:42.860475 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851457 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 01:15:42.860475 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851461 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 01:15:42.860475 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851463 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 01:15:42.860475 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851466 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 01:15:42.860475 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851469 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 01:15:42.860966 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851471 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 01:15:42.860966 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851474 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 01:15:42.860966 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851477 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 01:15:42.860966 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851479 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 01:15:42.860966 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851483 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 01:15:42.860966 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851487 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 01:15:42.860966 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851490 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 01:15:42.860966 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851492 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 01:15:42.860966 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851495 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 01:15:42.860966 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851497 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 01:15:42.860966 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851501 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 01:15:42.860966 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851505 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 01:15:42.860966 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851508 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 01:15:42.860966 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851511 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 01:15:42.860966 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851514 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 01:15:42.860966 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851516 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 01:15:42.860966 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851520 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 01:15:42.860966 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851522 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 01:15:42.860966 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851525 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 01:15:42.861443 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851528 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 01:15:42.861443 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851531 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 01:15:42.861443 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851534 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 01:15:42.861443 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851537 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 01:15:42.861443 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851539 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 01:15:42.861443 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851542 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 01:15:42.861443 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851547 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 01:15:42.861443 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851550 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 01:15:42.861443 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851553 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 01:15:42.861443 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851557 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 01:15:42.861443 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851559 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 01:15:42.861443 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851562 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 01:15:42.861443 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851565 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 01:15:42.861443 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851567 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 01:15:42.861443 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851570 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 01:15:42.861443 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851573 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 01:15:42.861443 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851575 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 01:15:42.861443 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851578 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 01:15:42.861443 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851581 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 01:15:42.861443 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851583 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 01:15:42.861936 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851586 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 01:15:42.861936 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851588 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 01:15:42.861936 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851591 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 01:15:42.861936 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851594 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 01:15:42.861936 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851596 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 01:15:42.861936 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851599 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 01:15:42.861936 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851602 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 01:15:42.861936 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851605 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 01:15:42.861936 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851607 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 01:15:42.861936 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851610 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 01:15:42.861936 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851612 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 01:15:42.861936 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851615 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 01:15:42.861936 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851617 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 01:15:42.861936 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851620 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 01:15:42.861936 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851623 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 01:15:42.861936 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851625 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 01:15:42.861936 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851628 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 01:15:42.861936 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851630 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 01:15:42.861936 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851634 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 01:15:42.861936 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851637 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 01:15:42.862444 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851640 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 01:15:42.862444 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.851645 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 01:15:42.862444 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.852214 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 01:15:42.862444 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.858845 2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 28 01:15:42.862444 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.858864 2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 28 01:15:42.862444 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.858884 2565 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 28 01:15:42.862444 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.858890 2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 28 01:15:42.862444 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.858949 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 01:15:42.862444 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.858954 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 01:15:42.862444 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.858957 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 01:15:42.862444 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.858960 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 01:15:42.862444 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.858963 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 01:15:42.862444 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.858966 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 01:15:42.862444 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.858969 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 01:15:42.862808 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.858972 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 01:15:42.862808 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.858974 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 01:15:42.862808 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.858977 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 01:15:42.862808 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.858980 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 01:15:42.862808 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.858983 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 01:15:42.862808 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.858986 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 01:15:42.862808 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.858988 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 01:15:42.862808 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.858991 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 01:15:42.862808 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.858994 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 01:15:42.862808 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.858996 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 01:15:42.862808 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.858998 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 01:15:42.862808 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859001 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 01:15:42.862808 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859003 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 01:15:42.862808 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859006 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 01:15:42.862808 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859009 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 01:15:42.862808 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859011 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 01:15:42.862808 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859014 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 01:15:42.862808 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859016 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 01:15:42.862808 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859019 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 01:15:42.862808 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859022 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 01:15:42.863341 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859025 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 01:15:42.863341 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859027 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 01:15:42.863341 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859054 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 01:15:42.863341 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859059 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 01:15:42.863341 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859063 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 01:15:42.863341 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859066 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 01:15:42.863341 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859070 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 01:15:42.863341 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859073 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 01:15:42.863341 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859077 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 01:15:42.863341 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859080 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 01:15:42.863341 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859083 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 01:15:42.863341 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859085 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 01:15:42.863341 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859088 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 01:15:42.863341 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859091 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 01:15:42.863341 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859094 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 01:15:42.863341 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859097 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 01:15:42.863341 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859105 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 01:15:42.863341 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859108 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 01:15:42.863341 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859111 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 01:15:42.863799 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859114 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 01:15:42.863799 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859117 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 01:15:42.863799 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859120 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 01:15:42.863799 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859123 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 01:15:42.863799 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859127 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 01:15:42.863799 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859130 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 01:15:42.863799 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859133 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 01:15:42.863799 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859136 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 01:15:42.863799 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859139 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 01:15:42.863799 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859141 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 01:15:42.863799 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859144 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 01:15:42.863799 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859147 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 01:15:42.863799 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859150 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 01:15:42.863799 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859152 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 01:15:42.863799 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859155 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 01:15:42.863799 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859158 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 01:15:42.863799 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859167 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 01:15:42.863799 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859169 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 01:15:42.863799 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859172 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 01:15:42.864285 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859174 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 01:15:42.864285 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859177 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 28 01:15:42.864285 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859180 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 01:15:42.864285 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859182 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 01:15:42.864285 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859184 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 01:15:42.864285 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859187 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 01:15:42.864285 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859189 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 01:15:42.864285 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859192 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 01:15:42.864285 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859194 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 01:15:42.864285 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859197 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 01:15:42.864285 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859199 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 01:15:42.864285 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859202 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 01:15:42.864285 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859204 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 01:15:42.864285 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859207 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 01:15:42.864285 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859210 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 01:15:42.864285 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859212 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 01:15:42.864285 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859215 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 01:15:42.864285 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859217 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 01:15:42.864285 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859220 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 01:15:42.864285 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859222 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 01:15:42.864775 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859224 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 01:15:42.864775 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859227 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 01:15:42.864775 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859229 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 01:15:42.864775 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859232 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 01:15:42.864775 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859235 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 01:15:42.864775 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859237 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 01:15:42.864775 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859240 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 01:15:42.864775 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859242 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 01:15:42.864775 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859246 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 01:15:42.864775 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859248 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 01:15:42.864775 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859257 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 01:15:42.864775 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859260 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 01:15:42.864775 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859262 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 01:15:42.864775 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859264 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 01:15:42.864775 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.859269 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 01:15:42.865198 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859407 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 01:15:42.865198 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859411 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 01:15:42.865198 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859414 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 01:15:42.865198 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859417 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 01:15:42.865198 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859420 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 01:15:42.865198 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859423 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 01:15:42.865198 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859426 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 01:15:42.865198 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859429 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 01:15:42.865198 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859432 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 01:15:42.865198 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859435 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 01:15:42.865198 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859437 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 01:15:42.865198 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859441 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 01:15:42.865198 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859444 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 01:15:42.865198 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859447 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 01:15:42.865198 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859450 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 01:15:42.865198 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859452 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 01:15:42.865198 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859455 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 01:15:42.865198 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859458 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 01:15:42.865198 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859462 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 01:15:42.865669 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859465 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 01:15:42.865669 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859468 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 01:15:42.865669 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859471 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 01:15:42.865669 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859474 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 01:15:42.865669 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859476 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 01:15:42.865669 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859479 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 01:15:42.865669 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859482 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 01:15:42.865669 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859485 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 01:15:42.865669 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859487 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 01:15:42.865669 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859495 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 01:15:42.865669 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859498 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 01:15:42.865669 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859500 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 01:15:42.865669 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859503 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 01:15:42.865669 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859505 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 01:15:42.865669 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859508 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 01:15:42.865669 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859510 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 01:15:42.865669 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859512 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 01:15:42.865669 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859515 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 01:15:42.865669 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859517 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 01:15:42.865669 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859520 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 28 01:15:42.866164 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859523 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 01:15:42.866164 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859525 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 01:15:42.866164 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859527 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 01:15:42.866164 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859530 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 01:15:42.866164 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859532 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 01:15:42.866164 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859534 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 01:15:42.866164 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859538 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 01:15:42.866164 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859542 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 01:15:42.866164 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859545 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 01:15:42.866164 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859548 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 01:15:42.866164 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859551 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 01:15:42.866164 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859553 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 01:15:42.866164 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859556 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 01:15:42.866164 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859559 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 01:15:42.866164 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859561 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 01:15:42.866164 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859564 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 01:15:42.866164 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859567 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 01:15:42.866164 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859570 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 01:15:42.866164 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859573 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 01:15:42.866630 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859576 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 01:15:42.866630 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859578 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 01:15:42.866630 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859581 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 01:15:42.866630 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859589 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 01:15:42.866630 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859592 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 01:15:42.866630 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859595 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 01:15:42.866630 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859597 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 01:15:42.866630 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859600 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 01:15:42.866630 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859603 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 01:15:42.866630 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859606 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 01:15:42.866630 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859608 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 01:15:42.866630 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859611 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 01:15:42.866630 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859614 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 01:15:42.866630 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859616 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 01:15:42.866630 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859619 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 01:15:42.866630 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859621 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 01:15:42.866630 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859624 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 01:15:42.866630 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859626 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 01:15:42.866630 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859629 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 01:15:42.866630 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859631 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 01:15:42.867124 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859634 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 01:15:42.867124 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859636 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 01:15:42.867124 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859638 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 01:15:42.867124 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859641 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 01:15:42.867124 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859643 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 01:15:42.867124 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859646 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 01:15:42.867124 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859648 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 01:15:42.867124 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859651 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 01:15:42.867124 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859654 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 01:15:42.867124 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859656 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 01:15:42.867124 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859659 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 01:15:42.867124 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859661 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 01:15:42.867124 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859664 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 01:15:42.867124 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859666 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 01:15:42.867124 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859669 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 01:15:42.867124 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859672 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 01:15:42.867124 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859679 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 01:15:42.867124 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859682 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 01:15:42.867124 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859684 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 01:15:42.867124 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859687 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 01:15:42.867633 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:42.859690 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 01:15:42.867633 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.859694 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 01:15:42.867633 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.859803 2565 server.go:962] "Client rotation is on, will bootstrap in background" Jul 28 01:15:42.867633 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.861974 2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 28 01:15:42.867633 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.863387 2565 server.go:1019] "Starting client certificate rotation" Jul 28 01:15:42.867633 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.863488 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 01:15:42.867633 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.864406 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 01:15:42.867633 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.866503 2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 01:15:42.867869 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.867771 2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 01:15:42.876999 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.876982 2565 log.go:25] "Validated CRI v1 runtime API" Jul 28 01:15:42.880402 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.880387 2565 log.go:25] "Validated CRI v1 image API" Jul 28 01:15:42.882042 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.882015 2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 28 01:15:42.885949 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.885928 2565 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7b53b500-f1ec-4265-9659-80515ae8e4c8:/dev/nvme0n1p3 ebe81ef8-78e8-499e-b8dd-a89efbd11516:/dev/nvme0n1p4] Jul 28 01:15:42.886020 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.885949 2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 28 01:15:42.890913 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.890887 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 01:15:42.891875 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.891775 2565 manager.go:216] Machine: {Timestamp:2026-07-28 01:15:42.889613534 +0000 UTC m=+0.324011839 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3125028 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25844eb80b2f0e1d6475b8a98603e4 SystemUUID:ec25844e-b80b-2f0e-1d64-75b8a98603e4 BootID:f58c8712-62d1-43d6-91b1-16f365c1a940 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:41:11:75:68:83 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:41:11:75:68:83 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:8b:ba:9a:58:e2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 28 01:15:42.891875 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.891870 2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 28 01:15:42.891980 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.891945 2565 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 28 01:15:42.894802 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.894781 2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 28 01:15:42.894935 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.894807 2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-23.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 28 01:15:42.895925 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.895915 2565 topology_manager.go:138] "Creating topology manager with none policy" Jul 28 01:15:42.895958 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.895928 2565 container_manager_linux.go:309] "Creating device plugin manager" Jul 28 01:15:42.895958 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.895937 2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 01:15:42.896567 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.896557 2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 01:15:42.898072 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.898062 2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 28 01:15:42.901125 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.901113 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 28 01:15:42.901249 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.901239 2565 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 28 01:15:42.901286 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.901277 2565 kubelet.go:486] "Attempting to sync node with API server" Jul 28 01:15:42.901319 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.901294 2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 28 01:15:42.901348 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.901323 2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 28 01:15:42.901379 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.901349 2565 kubelet.go:398] "Adding apiserver pod source" Jul 28 01:15:42.901379 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.901361 2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 28 01:15:42.901738 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.901727 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 01:15:42.901780 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.901743 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 01:15:42.903947 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.903933 2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 28 01:15:42.906239 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.906222 2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 28 01:15:42.906291 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.906255 2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 28 01:15:42.908180 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.908161 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 28 01:15:42.908251 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.908187 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 28 01:15:42.908251 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.908197 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 28 01:15:42.908251 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.908205 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 28 01:15:42.908251 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.908218 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 28 01:15:42.908251 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.908228 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 28 01:15:42.908251 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.908239 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 28 01:15:42.908499 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.908255 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 28 01:15:42.908499 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.908275 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 28 01:15:42.908499 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.908284 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 28 01:15:42.908499 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.908306 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 28 01:15:42.908499 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.908315 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 28 01:15:42.908499 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.908340 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 28 01:15:42.908499 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.908348 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 28 01:15:42.912072 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.912056 2565 server.go:1268] "Started kubelet" Jul 28 01:15:42.912183 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.912157 2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 28 01:15:42.912247 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.912136 2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 28 01:15:42.912247 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.912216 2565 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 28 01:15:42.912771 ip-10-0-131-23 systemd[1]: Started Kubernetes Kubelet. Jul 28 01:15:42.912963 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:42.912806 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-23.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 28 01:15:42.912963 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.912822 2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-23.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 28 01:15:42.912963 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:42.912808 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 28 01:15:42.912963 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.912954 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ckmw6" Jul 28 01:15:42.913318 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.913285 2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 28 01:15:42.914167 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.914151 2565 server.go:310] "Adding debug handlers to kubelet server" Jul 28 01:15:42.917127 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:42.917108 2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 28 01:15:42.920391 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.920371 2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 28 01:15:42.920391 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.920383 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 28 01:15:42.920548 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.920411 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ckmw6" Jul 28 01:15:42.920548 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:42.920432 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-23.ec2.internal\" not found" Jul 28 01:15:42.920940 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.920921 2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 28 01:15:42.920940 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.920925 2565 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 28 01:15:42.921101 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.920947 2565 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 28 01:15:42.921161 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.921118 2565 factory.go:55] Registering systemd factory Jul 28 01:15:42.921161 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.921132 2565 factory.go:223] Registration of the systemd container factory successfully Jul 28 01:15:42.921161 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:42.920186 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-23.ec2.internal.18c64f29b8d220a2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-23.ec2.internal,UID:ip-10-0-131-23.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-23.ec2.internal,},FirstTimestamp:2026-07-28 01:15:42.91201245 +0000 UTC m=+0.346410755,LastTimestamp:2026-07-28 01:15:42.91201245 +0000 UTC m=+0.346410755,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-23.ec2.internal,}" Jul 28 01:15:42.921343 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.921324 2565 factory.go:153] Registering CRI-O factory Jul 28 01:15:42.921343 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.921339 2565 factory.go:223] Registration of the crio container factory successfully Jul 28 01:15:42.921789 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.921388 2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 28 01:15:42.921789 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.921413 2565 factory.go:103] Registering Raw factory Jul 28 01:15:42.921789 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.921426 2565 manager.go:1196] Started watching for new ooms in manager Jul 28 01:15:42.921947 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.921921 2565 manager.go:349] Starting recovery of all containers Jul 28 01:15:42.922127 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.922113 2565 reconstruct.go:97] "Volume reconstruction finished" Jul 28 01:15:42.922127 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.922126 2565 reconciler.go:29] "Reconciler: start to sync state" Jul 28 01:15:42.925449 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:42.925421 2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-23.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 28 01:15:42.925554 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:42.925531 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 28 01:15:42.933598 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.933577 2565 manager.go:354] Recovery completed Jul 28 01:15:42.938957 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.938942 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 01:15:42.940576 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.940552 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-23.ec2.internal" event="NodeHasSufficientMemory" Jul 28 01:15:42.940660 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.940582 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-23.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 01:15:42.940660 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.940596 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-23.ec2.internal" event="NodeHasSufficientPID" Jul 28 01:15:42.941114 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.941101 2565 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 28 01:15:42.941114 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.941113 2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 28 01:15:42.941193 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.941128 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 28 01:15:42.941193 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.941155 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 01:15:42.941781 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.941768 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-23.ec2.internal" event="NodeHasSufficientMemory" Jul 28 01:15:42.941860 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.941795 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-23.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 01:15:42.941860 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.941808 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-23.ec2.internal" event="NodeHasSufficientPID" Jul 28 01:15:42.942752 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.942741 2565 policy_none.go:49] "None policy: Start" Jul 28 01:15:42.942795 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.942755 2565 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 28 01:15:42.942795 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.942765 2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 28 01:15:42.943758 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.943749 2565 policy_none.go:47] "Start" Jul 28 01:15:42.982688 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.982584 2565 manager.go:337] "Starting Device Plugin manager" Jul 28 01:15:42.982772 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:42.982701 2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 28 01:15:42.982772 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.982713 2565 server.go:87] "Starting device plugin registration server" Jul 28 01:15:42.982896 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.982886 2565 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 28 01:15:42.982943 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.982898 2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 28 01:15:42.983183 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.982996 2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 28 01:15:42.983183 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.983107 2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 28 01:15:42.983183 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:42.983117 2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 28 01:15:42.983596 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:42.983580 2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 28 01:15:42.983675 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:42.983618 2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-23.ec2.internal\" not found" Jul 28 01:15:43.083237 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.083212 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 01:15:43.084635 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.084063 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-23.ec2.internal" event="NodeHasSufficientMemory" Jul 28 01:15:43.084635 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.084090 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-23.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 01:15:43.084635 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.084107 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-23.ec2.internal" event="NodeHasSufficientPID" Jul 28 01:15:43.084635 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.084138 2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-23.ec2.internal" Jul 28 01:15:43.093342 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.093293 2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-23.ec2.internal" Jul 28 01:15:43.093342 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:43.093315 2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-23.ec2.internal\": node \"ip-10-0-131-23.ec2.internal\" not found" Jul 28 01:15:43.093564 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.093540 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 28 01:15:43.094886 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.094867 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 28 01:15:43.094977 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.094906 2565 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 28 01:15:43.094977 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.094949 2565 kubelet.go:2448] "Starting kubelet main sync loop" Jul 28 01:15:43.095093 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:43.094993 2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 28 01:15:43.097372 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.097352 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 01:15:43.139932 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:43.139909 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-23.ec2.internal\" not found" Jul 28 01:15:43.199104 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.199082 2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-23.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-23.ec2.internal"] Jul 28 01:15:43.199172 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.199157 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 01:15:43.200310 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.200291 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-23.ec2.internal" event="NodeHasSufficientMemory" Jul 28 01:15:43.200384 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.200323 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-23.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 01:15:43.200384 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.200335 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-23.ec2.internal" event="NodeHasSufficientPID" Jul 28 01:15:43.201853 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.201839 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 01:15:43.201957 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.201944 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-23.ec2.internal" Jul 28 01:15:43.201999 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.201972 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 01:15:43.202601 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.202586 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-23.ec2.internal" event="NodeHasSufficientMemory" Jul 28 01:15:43.202684 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.202610 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-23.ec2.internal" event="NodeHasSufficientMemory" Jul 28 01:15:43.202684 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.202630 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-23.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 01:15:43.202684 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.202641 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-23.ec2.internal" event="NodeHasSufficientPID" Jul 28 01:15:43.202684 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.202612 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-23.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 01:15:43.202830 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.202708 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-23.ec2.internal" event="NodeHasSufficientPID" Jul 28 01:15:43.203922 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.203908 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-23.ec2.internal" Jul 28 01:15:43.203960 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.203943 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 01:15:43.204557 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.204541 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-23.ec2.internal" event="NodeHasSufficientMemory" Jul 28 01:15:43.204642 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.204567 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-23.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 01:15:43.204642 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.204581 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-23.ec2.internal" event="NodeHasSufficientPID" Jul 28 01:15:43.224748 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.224724 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6a601e81bd2ad6ddab885bb95322a480-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-23.ec2.internal\" (UID: \"6a601e81bd2ad6ddab885bb95322a480\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-23.ec2.internal" Jul 28 01:15:43.224841 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.224771 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a601e81bd2ad6ddab885bb95322a480-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-23.ec2.internal\" (UID: \"6a601e81bd2ad6ddab885bb95322a480\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-23.ec2.internal" Jul 28 01:15:43.224841 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.224788 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3ff7e6d6dc372649b853c14fc595bb8f-config\") pod \"kube-apiserver-proxy-ip-10-0-131-23.ec2.internal\" (UID: \"3ff7e6d6dc372649b853c14fc595bb8f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-23.ec2.internal" Jul 28 01:15:43.226975 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:43.226939 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-23.ec2.internal\" not found" node="ip-10-0-131-23.ec2.internal" Jul 28 01:15:43.231747 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:43.231732 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-23.ec2.internal\" not found" node="ip-10-0-131-23.ec2.internal" Jul 28 01:15:43.240453 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:43.240440 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-23.ec2.internal\" not found" Jul 28 01:15:43.325197 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.325169 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6a601e81bd2ad6ddab885bb95322a480-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-23.ec2.internal\" (UID: \"6a601e81bd2ad6ddab885bb95322a480\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-23.ec2.internal" Jul 28 01:15:43.325197 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.325197 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a601e81bd2ad6ddab885bb95322a480-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-23.ec2.internal\" (UID: \"6a601e81bd2ad6ddab885bb95322a480\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-23.ec2.internal" Jul 28 01:15:43.325388 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.325216 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3ff7e6d6dc372649b853c14fc595bb8f-config\") pod \"kube-apiserver-proxy-ip-10-0-131-23.ec2.internal\" (UID: \"3ff7e6d6dc372649b853c14fc595bb8f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-23.ec2.internal" Jul 28 01:15:43.325388 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.325244 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3ff7e6d6dc372649b853c14fc595bb8f-config\") pod \"kube-apiserver-proxy-ip-10-0-131-23.ec2.internal\" (UID: \"3ff7e6d6dc372649b853c14fc595bb8f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-23.ec2.internal" Jul 28 01:15:43.325388 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.325270 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6a601e81bd2ad6ddab885bb95322a480-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-23.ec2.internal\" (UID: \"6a601e81bd2ad6ddab885bb95322a480\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-23.ec2.internal" Jul 28 01:15:43.325388 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.325273 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a601e81bd2ad6ddab885bb95322a480-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-23.ec2.internal\" (UID: \"6a601e81bd2ad6ddab885bb95322a480\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-23.ec2.internal" Jul 28 01:15:43.341012 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:43.340966 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-23.ec2.internal\" not found" Jul 28 01:15:43.441214 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:43.441186 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-23.ec2.internal\" not found" Jul 28 01:15:43.529312 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.529277 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-23.ec2.internal" Jul 28 01:15:43.536258 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.536240 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-23.ec2.internal" Jul 28 01:15:43.541869 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:43.541850 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-23.ec2.internal\" not found" Jul 28 01:15:43.642192 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:43.642160 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-23.ec2.internal\" not found" Jul 28 01:15:43.743443 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:43.743374 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-23.ec2.internal\" not found" Jul 28 01:15:43.844431 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:43.844394 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-23.ec2.internal\" not found" Jul 28 01:15:43.862506 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.862479 2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 28 01:15:43.862672 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.862653 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 01:15:43.926348 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.926318 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 01:10:43 +0000 UTC" deadline="2028-02-07 12:56:24.436418332 +0000 UTC" Jul 28 01:15:43.926348 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.926346 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13427h40m40.510076119s" Jul 28 01:15:43.945482 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:43.945455 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-23.ec2.internal\" not found" Jul 28 01:15:43.952466 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:43.952446 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 01:15:44.046220 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:44.046138 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-23.ec2.internal\" not found" Jul 28 01:15:44.146599 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:44.146577 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-23.ec2.internal\" not found" Jul 28 01:15:44.244715 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:44.244683 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff7e6d6dc372649b853c14fc595bb8f.slice/crio-985ee986b55cac745d1017c6b1dfe995285f506c26a8d20b1383ef1677231403 WatchSource:0}: Error finding container 985ee986b55cac745d1017c6b1dfe995285f506c26a8d20b1383ef1677231403: Status 404 returned error can't find the container with id 985ee986b55cac745d1017c6b1dfe995285f506c26a8d20b1383ef1677231403 Jul 28 01:15:44.245512 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:44.245489 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a601e81bd2ad6ddab885bb95322a480.slice/crio-a95487d7fab12d8cef29b993cc6d528e0969be6248a0a9e57572dc0ffa7b3bd5 WatchSource:0}: Error finding container a95487d7fab12d8cef29b993cc6d528e0969be6248a0a9e57572dc0ffa7b3bd5: Status 404 returned error can't find the container with id a95487d7fab12d8cef29b993cc6d528e0969be6248a0a9e57572dc0ffa7b3bd5 Jul 28 01:15:44.247720 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:44.247702 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-23.ec2.internal\" not found" Jul 28 01:15:44.250579 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.250565 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 01:15:44.283239 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.283218 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 01:15:44.323274 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.323230 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-23.ec2.internal" Jul 28 01:15:44.334285 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.334269 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 01:15:44.336675 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.336662 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-23.ec2.internal" Jul 28 01:15:44.348066 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.348046 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 01:15:44.473443 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.473417 2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 01:15:44.902567 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.902527 2565 apiserver.go:52] "Watching apiserver" Jul 28 01:15:44.909421 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.909390 2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 28 01:15:44.911141 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.911090 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-2trzf","openshift-network-operator/iptables-alerter-sf4nf","openshift-ovn-kubernetes/ovnkube-node-kbl47","kube-system/konnectivity-agent-xtl8m","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldj2w","openshift-cluster-node-tuning-operator/tuned-xjxrs","openshift-multus/multus-rcvxw","openshift-multus/network-metrics-daemon-zwr5s","openshift-network-diagnostics/network-check-target-6swc2","kube-system/global-pull-secret-syncer-8sv4q","kube-system/kube-apiserver-proxy-ip-10-0-131-23.ec2.internal","openshift-dns/node-resolver-snwqc","openshift-image-registry/node-ca-zlwvl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-23.ec2.internal"] Jul 28 01:15:44.912968 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.912949 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rcvxw" Jul 28 01:15:44.914497 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.914468 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zlwvl" Jul 28 01:15:44.915725 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.915673 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 28 01:15:44.916237 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.916220 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:44.917212 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.916813 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 28 01:15:44.917212 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.916868 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nrm52\"" Jul 28 01:15:44.917212 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.916929 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 28 01:15:44.917212 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.916936 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 28 01:15:44.917212 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.917013 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 28 01:15:44.917212 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.917013 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 28 01:15:44.917212 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.917032 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-55544\"" Jul 28 01:15:44.917741 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.917723 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 28 01:15:44.917812 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.917756 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xtl8m" Jul 28 01:15:44.918619 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.918599 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 28 01:15:44.919820 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.918978 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 28 01:15:44.919820 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.919243 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kct8j\"" Jul 28 01:15:44.919820 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.919294 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 28 01:15:44.919820 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.919381 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 28 01:15:44.919820 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.919585 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 28 01:15:44.919820 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.919804 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 28 01:15:44.920203 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.919838 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldj2w" Jul 28 01:15:44.920412 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.920398 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qzh6m\"" Jul 28 01:15:44.922212 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.920923 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 28 01:15:44.922212 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.921146 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 28 01:15:44.922212 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.921487 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:44.922212 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.921635 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 28 01:15:44.922576 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.922548 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 28 01:15:44.923053 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.923002 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6swc2" Jul 28 01:15:44.923969 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:44.923932 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6swc2" podUID="bcb4dce7-214c-4f64-b5af-87ca7b0d6170" Jul 28 01:15:44.924449 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.924427 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pbrmg\"" Jul 28 01:15:44.924736 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.924713 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 28 01:15:44.925004 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.924987 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 28 01:15:44.928129 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.926265 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 28 01:15:44.928129 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.926403 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5b2mx\"" Jul 28 01:15:44.928129 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.926269 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 28 01:15:44.929737 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.929715 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2trzf" Jul 28 01:15:44.932300 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.932279 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sf4nf" Jul 28 01:15:44.932990 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.932972 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-skb55\"" Jul 28 01:15:44.933424 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.933406 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv4q" Jul 28 01:15:44.933515 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:44.933489 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8sv4q" podUID="12e2c22a-bb31-4f97-869a-3c89eb64f837" Jul 28 01:15:44.933879 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.933854 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 28 01:15:44.934507 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.934482 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-run-systemd\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:44.934601 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.934520 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6db0343-a5f9-42ad-9869-539525bd315a-sys\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:44.934601 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.934592 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-multus-cni-dir\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:44.934714 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.934616 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-multus-conf-dir\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:44.934714 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.934641 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w46w\" (UniqueName: \"kubernetes.io/projected/07541f83-addd-4996-9a31-74860f8e3ba3-kube-api-access-6w46w\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:44.934714 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.934667 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a29b525f-58ac-47f8-b759-d81742c281f4-cni-binary-copy\") pod \"multus-additional-cni-plugins-2trzf\" (UID: \"a29b525f-58ac-47f8-b759-d81742c281f4\") " pod="openshift-multus/multus-additional-cni-plugins-2trzf" Jul 28 01:15:44.934714 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.934692 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-host-slash\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:44.934919 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.934715 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b6db0343-a5f9-42ad-9869-539525bd315a-etc-tuned\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:44.934919 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.934740 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh9gl\" (UniqueName: \"kubernetes.io/projected/b6db0343-a5f9-42ad-9869-539525bd315a-kube-api-access-fh9gl\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:44.934919 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.934765 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-host-kubelet\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:44.934919 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.934790 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-host-run-ovn-kubernetes\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:44.934919 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.934813 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/92b25e3a-2150-47b0-b6e4-437e77a20172-socket-dir\") pod \"aws-ebs-csi-driver-node-ldj2w\" (UID: \"92b25e3a-2150-47b0-b6e4-437e77a20172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldj2w" Jul 28 01:15:44.934919 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.934838 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/92b25e3a-2150-47b0-b6e4-437e77a20172-etc-selinux\") pod \"aws-ebs-csi-driver-node-ldj2w\" (UID: \"92b25e3a-2150-47b0-b6e4-437e77a20172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldj2w" Jul 28 01:15:44.934919 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.934864 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-multus-socket-dir-parent\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:44.934919 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.934887 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ltvc\" (UniqueName: \"kubernetes.io/projected/f32e8846-0caa-4b52-a5fd-ae65659fdea1-kube-api-access-7ltvc\") pod \"node-ca-zlwvl\" (UID: \"f32e8846-0caa-4b52-a5fd-ae65659fdea1\") " pod="openshift-image-registry/node-ca-zlwvl" Jul 28 01:15:44.935329 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.934927 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-run-openvswitch\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:44.935329 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.934952 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-cnibin\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:44.935329 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.934988 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-host-var-lib-kubelet\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:44.935329 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.935014 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-host-cni-bin\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:44.935329 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.935040 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d018a93b-4db3-45cb-8e11-919a45a35718-agent-certs\") pod \"konnectivity-agent-xtl8m\" (UID: \"d018a93b-4db3-45cb-8e11-919a45a35718\") " pod="kube-system/konnectivity-agent-xtl8m" Jul 28 01:15:44.935329 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.935104 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b6db0343-a5f9-42ad-9869-539525bd315a-etc-sysctl-d\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:44.935329 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.935152 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6db0343-a5f9-42ad-9869-539525bd315a-tmp\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:44.935329 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.935177 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-host-var-lib-cni-bin\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:44.935329 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.935211 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-log-socket\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:44.935329 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.935325 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-ovnkube-config\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:44.935826 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.935389 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d018a93b-4db3-45cb-8e11-919a45a35718-konnectivity-ca\") pod \"konnectivity-agent-xtl8m\" (UID: \"d018a93b-4db3-45cb-8e11-919a45a35718\") " pod="kube-system/konnectivity-agent-xtl8m" Jul 28 01:15:44.935826 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.935429 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-host-run-netns\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:44.935826 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.935468 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-hostroot\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:44.935826 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.935510 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 28 01:15:44.935826 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.935516 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-run-ovn\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:44.935826 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.935542 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-host-cni-netd\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:44.935826 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.935611 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/92b25e3a-2150-47b0-b6e4-437e77a20172-registration-dir\") pod \"aws-ebs-csi-driver-node-ldj2w\" (UID: \"92b25e3a-2150-47b0-b6e4-437e77a20172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldj2w" Jul 28 01:15:44.935826 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.935639 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b6db0343-a5f9-42ad-9869-539525bd315a-host\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:44.935826 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.935643 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 28 01:15:44.935826 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.935514 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 28 01:15:44.935826 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.935665 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a29b525f-58ac-47f8-b759-d81742c281f4-system-cni-dir\") pod \"multus-additional-cni-plugins-2trzf\" (UID: \"a29b525f-58ac-47f8-b759-d81742c281f4\") " pod="openshift-multus/multus-additional-cni-plugins-2trzf" Jul 28 01:15:44.935826 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.935691 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a29b525f-58ac-47f8-b759-d81742c281f4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2trzf\" (UID: \"a29b525f-58ac-47f8-b759-d81742c281f4\") " pod="openshift-multus/multus-additional-cni-plugins-2trzf" Jul 28 01:15:44.935826 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.935730 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb797\" (UniqueName: \"kubernetes.io/projected/a29b525f-58ac-47f8-b759-d81742c281f4-kube-api-access-mb797\") pod \"multus-additional-cni-plugins-2trzf\" (UID: \"a29b525f-58ac-47f8-b759-d81742c281f4\") " pod="openshift-multus/multus-additional-cni-plugins-2trzf" Jul 28 01:15:44.935826 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.935755 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-host-run-netns\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:44.935826 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.935780 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbdjv\" (UniqueName: \"kubernetes.io/projected/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-kube-api-access-vbdjv\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:44.935826 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.935808 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b6db0343-a5f9-42ad-9869-539525bd315a-etc-modprobe-d\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:44.936520 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.935856 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-host-var-lib-cni-multus\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:44.936520 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.935900 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9l7l6\"" Jul 28 01:15:44.936520 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.935924 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-var-lib-openvswitch\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:44.936520 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.935948 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:44.936520 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.935971 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6db0343-a5f9-42ad-9869-539525bd315a-etc-kubernetes\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:44.936520 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.935996 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a29b525f-58ac-47f8-b759-d81742c281f4-os-release\") pod \"multus-additional-cni-plugins-2trzf\" (UID: \"a29b525f-58ac-47f8-b759-d81742c281f4\") " pod="openshift-multus/multus-additional-cni-plugins-2trzf" Jul 28 01:15:44.936520 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.936015 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 28 01:15:44.936520 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.936020 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-os-release\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:44.936520 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.936132 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhxk4\" (UniqueName: \"kubernetes.io/projected/bcb4dce7-214c-4f64-b5af-87ca7b0d6170-kube-api-access-hhxk4\") pod \"network-check-target-6swc2\" (UID: \"bcb4dce7-214c-4f64-b5af-87ca7b0d6170\") " pod="openshift-network-diagnostics/network-check-target-6swc2" Jul 28 01:15:44.936520 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.936170 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a29b525f-58ac-47f8-b759-d81742c281f4-cnibin\") pod \"multus-additional-cni-plugins-2trzf\" (UID: \"a29b525f-58ac-47f8-b759-d81742c281f4\") " pod="openshift-multus/multus-additional-cni-plugins-2trzf" Jul 28 01:15:44.936520 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.936199 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-etc-openvswitch\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:44.936520 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.936223 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-node-log\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:44.936520 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.936252 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-ovn-node-metrics-cert\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:44.936520 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.936278 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b6db0343-a5f9-42ad-9869-539525bd315a-etc-systemd\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:44.936520 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.936300 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b6db0343-a5f9-42ad-9869-539525bd315a-run\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:44.936520 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.936334 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07541f83-addd-4996-9a31-74860f8e3ba3-cni-binary-copy\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:44.936520 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.936357 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f32e8846-0caa-4b52-a5fd-ae65659fdea1-serviceca\") pod \"node-ca-zlwvl\" (UID: \"f32e8846-0caa-4b52-a5fd-ae65659fdea1\") " pod="openshift-image-registry/node-ca-zlwvl" Jul 28 01:15:44.937323 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.936381 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-env-overrides\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:44.937323 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.936409 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-ovnkube-script-lib\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:44.937323 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.936433 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6db0343-a5f9-42ad-9869-539525bd315a-var-lib-kubelet\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:44.937323 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.936471 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b6db0343-a5f9-42ad-9869-539525bd315a-etc-sysctl-conf\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:44.937323 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.936504 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-system-cni-dir\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:44.937323 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.936560 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a29b525f-58ac-47f8-b759-d81742c281f4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2trzf\" (UID: \"a29b525f-58ac-47f8-b759-d81742c281f4\") " pod="openshift-multus/multus-additional-cni-plugins-2trzf" Jul 28 01:15:44.937323 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.936604 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f32e8846-0caa-4b52-a5fd-ae65659fdea1-host\") pod \"node-ca-zlwvl\" (UID: \"f32e8846-0caa-4b52-a5fd-ae65659fdea1\") " pod="openshift-image-registry/node-ca-zlwvl" Jul 28 01:15:44.937323 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.936647 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/92b25e3a-2150-47b0-b6e4-437e77a20172-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ldj2w\" (UID: \"92b25e3a-2150-47b0-b6e4-437e77a20172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldj2w" Jul 28 01:15:44.937323 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.936675 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/92b25e3a-2150-47b0-b6e4-437e77a20172-device-dir\") pod \"aws-ebs-csi-driver-node-ldj2w\" (UID: \"92b25e3a-2150-47b0-b6e4-437e77a20172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldj2w" Jul 28 01:15:44.937323 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.936712 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9ln8\" (UniqueName: \"kubernetes.io/projected/92b25e3a-2150-47b0-b6e4-437e77a20172-kube-api-access-f9ln8\") pod \"aws-ebs-csi-driver-node-ldj2w\" (UID: \"92b25e3a-2150-47b0-b6e4-437e77a20172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldj2w" Jul 28 01:15:44.937323 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.936738 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/92b25e3a-2150-47b0-b6e4-437e77a20172-sys-fs\") pod \"aws-ebs-csi-driver-node-ldj2w\" (UID: \"92b25e3a-2150-47b0-b6e4-437e77a20172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldj2w" Jul 28 01:15:44.937323 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.936761 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b6db0343-a5f9-42ad-9869-539525bd315a-etc-sysconfig\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:44.937323 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.936785 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b6db0343-a5f9-42ad-9869-539525bd315a-lib-modules\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:44.937323 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.936823 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a29b525f-58ac-47f8-b759-d81742c281f4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2trzf\" (UID: \"a29b525f-58ac-47f8-b759-d81742c281f4\") " pod="openshift-multus/multus-additional-cni-plugins-2trzf" Jul 28 01:15:44.937323 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.936859 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/07541f83-addd-4996-9a31-74860f8e3ba3-multus-daemon-config\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:44.937323 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.936886 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-etc-kubernetes\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:44.938213 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.936913 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-systemd-units\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:44.938213 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.936936 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-host-run-k8s-cni-cncf-io\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:44.938213 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.936964 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-host-run-multus-certs\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:44.938213 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.938155 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 01:15:44.939284 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.939066 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwr5s" Jul 28 01:15:44.939284 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:44.939182 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwr5s" podUID="3308b659-80af-41d9-a040-f056900489b3" Jul 28 01:15:44.941249 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.941229 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-snwqc" Jul 28 01:15:44.944593 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.944573 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tfh7p\"" Jul 28 01:15:44.945162 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.945013 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 28 01:15:44.945162 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.945024 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 28 01:15:44.956321 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.956296 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xjhgw" Jul 28 01:15:44.963417 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:44.963071 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xjhgw" Jul 28 01:15:45.007198 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.007174 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 01:15:45.022182 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.022161 2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 28 01:15:45.037704 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.037646 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-ovn-node-metrics-cert\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.037704 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.037682 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b6db0343-a5f9-42ad-9869-539525bd315a-etc-systemd\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:45.037872 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.037710 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b6db0343-a5f9-42ad-9869-539525bd315a-run\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:45.037872 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.037735 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07541f83-addd-4996-9a31-74860f8e3ba3-cni-binary-copy\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:45.037872 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.037759 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f32e8846-0caa-4b52-a5fd-ae65659fdea1-serviceca\") pod \"node-ca-zlwvl\" (UID: \"f32e8846-0caa-4b52-a5fd-ae65659fdea1\") " pod="openshift-image-registry/node-ca-zlwvl" Jul 28 01:15:45.037872 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.037783 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-env-overrides\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.037872 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.037809 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-ovnkube-script-lib\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.037872 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.037811 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b6db0343-a5f9-42ad-9869-539525bd315a-etc-systemd\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:45.038484 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.037883 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b6db0343-a5f9-42ad-9869-539525bd315a-run\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:45.038484 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.037926 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6db0343-a5f9-42ad-9869-539525bd315a-var-lib-kubelet\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:45.038484 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.037954 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3308b659-80af-41d9-a040-f056900489b3-metrics-certs\") pod \"network-metrics-daemon-zwr5s\" (UID: \"3308b659-80af-41d9-a040-f056900489b3\") " pod="openshift-multus/network-metrics-daemon-zwr5s" Jul 28 01:15:45.038484 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.037980 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b6db0343-a5f9-42ad-9869-539525bd315a-etc-sysctl-conf\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:45.038484 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.038005 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-system-cni-dir\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:45.038484 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.038030 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a97a9823-8b73-453c-9d16-8907cc6b5116-hosts-file\") pod \"node-resolver-snwqc\" (UID: \"a97a9823-8b73-453c-9d16-8907cc6b5116\") " pod="openshift-dns/node-resolver-snwqc" Jul 28 01:15:45.038484 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.038059 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a29b525f-58ac-47f8-b759-d81742c281f4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2trzf\" (UID: \"a29b525f-58ac-47f8-b759-d81742c281f4\") " pod="openshift-multus/multus-additional-cni-plugins-2trzf" Jul 28 01:15:45.038484 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.038100 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f32e8846-0caa-4b52-a5fd-ae65659fdea1-host\") pod \"node-ca-zlwvl\" (UID: \"f32e8846-0caa-4b52-a5fd-ae65659fdea1\") " pod="openshift-image-registry/node-ca-zlwvl" Jul 28 01:15:45.038484 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.038127 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/92b25e3a-2150-47b0-b6e4-437e77a20172-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ldj2w\" (UID: \"92b25e3a-2150-47b0-b6e4-437e77a20172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldj2w" Jul 28 01:15:45.038484 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.038158 2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 28 01:15:45.038484 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.038190 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/92b25e3a-2150-47b0-b6e4-437e77a20172-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ldj2w\" (UID: \"92b25e3a-2150-47b0-b6e4-437e77a20172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldj2w" Jul 28 01:15:45.038484 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.038394 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-env-overrides\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.038484 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.038490 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f32e8846-0caa-4b52-a5fd-ae65659fdea1-serviceca\") pod \"node-ca-zlwvl\" (UID: \"f32e8846-0caa-4b52-a5fd-ae65659fdea1\") " pod="openshift-image-registry/node-ca-zlwvl" Jul 28 01:15:45.039162 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.038508 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07541f83-addd-4996-9a31-74860f8e3ba3-cni-binary-copy\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:45.039162 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.038521 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6db0343-a5f9-42ad-9869-539525bd315a-var-lib-kubelet\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:45.039162 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.038569 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-ovnkube-script-lib\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.039162 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.038596 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f32e8846-0caa-4b52-a5fd-ae65659fdea1-host\") pod \"node-ca-zlwvl\" (UID: \"f32e8846-0caa-4b52-a5fd-ae65659fdea1\") " pod="openshift-image-registry/node-ca-zlwvl" Jul 28 01:15:45.039162 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.038639 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/92b25e3a-2150-47b0-b6e4-437e77a20172-device-dir\") pod \"aws-ebs-csi-driver-node-ldj2w\" (UID: \"92b25e3a-2150-47b0-b6e4-437e77a20172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldj2w" Jul 28 01:15:45.039162 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.038665 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-system-cni-dir\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:45.039162 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.038672 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9ln8\" (UniqueName: \"kubernetes.io/projected/92b25e3a-2150-47b0-b6e4-437e77a20172-kube-api-access-f9ln8\") pod \"aws-ebs-csi-driver-node-ldj2w\" (UID: \"92b25e3a-2150-47b0-b6e4-437e77a20172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldj2w" Jul 28 01:15:45.039162 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.038713 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b6db0343-a5f9-42ad-9869-539525bd315a-etc-sysctl-conf\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:45.039162 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.038721 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfz76\" (UniqueName: \"kubernetes.io/projected/ac3e75a4-c6a5-4b7f-a767-08bcc895d5af-kube-api-access-xfz76\") pod \"iptables-alerter-sf4nf\" (UID: \"ac3e75a4-c6a5-4b7f-a767-08bcc895d5af\") " pod="openshift-network-operator/iptables-alerter-sf4nf" Jul 28 01:15:45.039162 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.038728 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/92b25e3a-2150-47b0-b6e4-437e77a20172-device-dir\") pod \"aws-ebs-csi-driver-node-ldj2w\" (UID: \"92b25e3a-2150-47b0-b6e4-437e77a20172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldj2w" Jul 28 01:15:45.039162 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.038754 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/92b25e3a-2150-47b0-b6e4-437e77a20172-sys-fs\") pod \"aws-ebs-csi-driver-node-ldj2w\" (UID: \"92b25e3a-2150-47b0-b6e4-437e77a20172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldj2w" Jul 28 01:15:45.039162 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.038794 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b6db0343-a5f9-42ad-9869-539525bd315a-etc-sysconfig\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:45.039162 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.038807 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a29b525f-58ac-47f8-b759-d81742c281f4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2trzf\" (UID: \"a29b525f-58ac-47f8-b759-d81742c281f4\") " pod="openshift-multus/multus-additional-cni-plugins-2trzf" Jul 28 01:15:45.039162 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.038822 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b6db0343-a5f9-42ad-9869-539525bd315a-lib-modules\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:45.039162 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.038831 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/92b25e3a-2150-47b0-b6e4-437e77a20172-sys-fs\") pod \"aws-ebs-csi-driver-node-ldj2w\" (UID: \"92b25e3a-2150-47b0-b6e4-437e77a20172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldj2w" Jul 28 01:15:45.039162 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.038850 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a29b525f-58ac-47f8-b759-d81742c281f4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2trzf\" (UID: \"a29b525f-58ac-47f8-b759-d81742c281f4\") " pod="openshift-multus/multus-additional-cni-plugins-2trzf" Jul 28 01:15:45.039162 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.038873 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b6db0343-a5f9-42ad-9869-539525bd315a-etc-sysconfig\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:45.040031 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.038877 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/07541f83-addd-4996-9a31-74860f8e3ba3-multus-daemon-config\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:45.040031 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.038914 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ac3e75a4-c6a5-4b7f-a767-08bcc895d5af-iptables-alerter-script\") pod \"iptables-alerter-sf4nf\" (UID: \"ac3e75a4-c6a5-4b7f-a767-08bcc895d5af\") " pod="openshift-network-operator/iptables-alerter-sf4nf" Jul 28 01:15:45.040031 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.038941 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a97a9823-8b73-453c-9d16-8907cc6b5116-tmp-dir\") pod \"node-resolver-snwqc\" (UID: \"a97a9823-8b73-453c-9d16-8907cc6b5116\") " pod="openshift-dns/node-resolver-snwqc" Jul 28 01:15:45.040031 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.038966 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-etc-kubernetes\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:45.040031 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.038989 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b6db0343-a5f9-42ad-9869-539525bd315a-lib-modules\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:45.040031 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.038991 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5ppq\" (UniqueName: \"kubernetes.io/projected/a97a9823-8b73-453c-9d16-8907cc6b5116-kube-api-access-p5ppq\") pod \"node-resolver-snwqc\" (UID: \"a97a9823-8b73-453c-9d16-8907cc6b5116\") " pod="openshift-dns/node-resolver-snwqc" Jul 28 01:15:45.040031 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.039039 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-etc-kubernetes\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:45.040031 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.039065 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-systemd-units\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.040031 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.039101 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-host-run-k8s-cni-cncf-io\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:45.040031 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.039144 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-host-run-multus-certs\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:45.040031 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.039147 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-systemd-units\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.040031 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.039200 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-host-run-k8s-cni-cncf-io\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:45.040031 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.039207 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-host-run-multus-certs\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:45.040031 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.039233 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-run-systemd\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.040031 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.039258 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6db0343-a5f9-42ad-9869-539525bd315a-sys\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:45.040031 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.039271 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-run-systemd\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.040031 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.039283 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-multus-cni-dir\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:45.040031 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.039307 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-multus-conf-dir\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:45.040942 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.039327 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6db0343-a5f9-42ad-9869-539525bd315a-sys\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:45.040942 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.039344 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w46w\" (UniqueName: \"kubernetes.io/projected/07541f83-addd-4996-9a31-74860f8e3ba3-kube-api-access-6w46w\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:45.040942 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.039369 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-multus-conf-dir\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:45.040942 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.039388 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/07541f83-addd-4996-9a31-74860f8e3ba3-multus-daemon-config\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:45.040942 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.039413 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac3e75a4-c6a5-4b7f-a767-08bcc895d5af-host-slash\") pod \"iptables-alerter-sf4nf\" (UID: \"ac3e75a4-c6a5-4b7f-a767-08bcc895d5af\") " pod="openshift-network-operator/iptables-alerter-sf4nf" Jul 28 01:15:45.040942 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.039463 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a29b525f-58ac-47f8-b759-d81742c281f4-cni-binary-copy\") pod \"multus-additional-cni-plugins-2trzf\" (UID: \"a29b525f-58ac-47f8-b759-d81742c281f4\") " pod="openshift-multus/multus-additional-cni-plugins-2trzf" Jul 28 01:15:45.040942 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.039477 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-multus-cni-dir\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:45.040942 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.039489 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-host-slash\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.040942 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.039479 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a29b525f-58ac-47f8-b759-d81742c281f4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2trzf\" (UID: \"a29b525f-58ac-47f8-b759-d81742c281f4\") " pod="openshift-multus/multus-additional-cni-plugins-2trzf" Jul 28 01:15:45.040942 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.039513 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b6db0343-a5f9-42ad-9869-539525bd315a-etc-tuned\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:45.040942 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.040126 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh9gl\" (UniqueName: \"kubernetes.io/projected/b6db0343-a5f9-42ad-9869-539525bd315a-kube-api-access-fh9gl\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:45.040942 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.040223 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/12e2c22a-bb31-4f97-869a-3c89eb64f837-kubelet-config\") pod \"global-pull-secret-syncer-8sv4q\" (UID: \"12e2c22a-bb31-4f97-869a-3c89eb64f837\") " pod="kube-system/global-pull-secret-syncer-8sv4q" Jul 28 01:15:45.040942 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.040273 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-host-kubelet\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.040942 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.040309 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-host-run-ovn-kubernetes\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.040942 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.040445 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/92b25e3a-2150-47b0-b6e4-437e77a20172-socket-dir\") pod \"aws-ebs-csi-driver-node-ldj2w\" (UID: \"92b25e3a-2150-47b0-b6e4-437e77a20172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldj2w" Jul 28 01:15:45.040942 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.040484 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/92b25e3a-2150-47b0-b6e4-437e77a20172-etc-selinux\") pod \"aws-ebs-csi-driver-node-ldj2w\" (UID: \"92b25e3a-2150-47b0-b6e4-437e77a20172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldj2w" Jul 28 01:15:45.040942 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.040522 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-multus-socket-dir-parent\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:45.041817 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.040551 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12e2c22a-bb31-4f97-869a-3c89eb64f837-original-pull-secret\") pod \"global-pull-secret-syncer-8sv4q\" (UID: \"12e2c22a-bb31-4f97-869a-3c89eb64f837\") " pod="kube-system/global-pull-secret-syncer-8sv4q" Jul 28 01:15:45.041817 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.040615 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ltvc\" (UniqueName: \"kubernetes.io/projected/f32e8846-0caa-4b52-a5fd-ae65659fdea1-kube-api-access-7ltvc\") pod \"node-ca-zlwvl\" (UID: \"f32e8846-0caa-4b52-a5fd-ae65659fdea1\") " pod="openshift-image-registry/node-ca-zlwvl" Jul 28 01:15:45.041817 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.040649 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-run-openvswitch\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.041817 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.040817 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-cnibin\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:45.041817 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.040855 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-host-var-lib-kubelet\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:45.041817 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.040893 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/12e2c22a-bb31-4f97-869a-3c89eb64f837-dbus\") pod \"global-pull-secret-syncer-8sv4q\" (UID: \"12e2c22a-bb31-4f97-869a-3c89eb64f837\") " pod="kube-system/global-pull-secret-syncer-8sv4q" Jul 28 01:15:45.041817 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.040940 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-host-cni-bin\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.041817 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.040986 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d018a93b-4db3-45cb-8e11-919a45a35718-agent-certs\") pod \"konnectivity-agent-xtl8m\" (UID: \"d018a93b-4db3-45cb-8e11-919a45a35718\") " pod="kube-system/konnectivity-agent-xtl8m" Jul 28 01:15:45.041817 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.041018 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b6db0343-a5f9-42ad-9869-539525bd315a-etc-sysctl-d\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:45.041817 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.041047 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6db0343-a5f9-42ad-9869-539525bd315a-tmp\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:45.041817 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.041127 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-host-var-lib-cni-bin\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:45.041817 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.041162 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-log-socket\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.041817 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.041219 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-ovnkube-config\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.041817 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.041252 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d018a93b-4db3-45cb-8e11-919a45a35718-konnectivity-ca\") pod \"konnectivity-agent-xtl8m\" (UID: \"d018a93b-4db3-45cb-8e11-919a45a35718\") " pod="kube-system/konnectivity-agent-xtl8m" Jul 28 01:15:45.041817 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.041280 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-host-run-netns\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:45.041817 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.041343 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-hostroot\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:45.041817 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.041430 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-hostroot\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:45.041817 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.039554 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-host-slash\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.042765 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.041937 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a29b525f-58ac-47f8-b759-d81742c281f4-cni-binary-copy\") pod \"multus-additional-cni-plugins-2trzf\" (UID: \"a29b525f-58ac-47f8-b759-d81742c281f4\") " pod="openshift-multus/multus-additional-cni-plugins-2trzf" Jul 28 01:15:45.042765 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.042626 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-host-kubelet\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.042765 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.042692 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-host-run-ovn-kubernetes\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.042928 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.042813 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/92b25e3a-2150-47b0-b6e4-437e77a20172-socket-dir\") pod \"aws-ebs-csi-driver-node-ldj2w\" (UID: \"92b25e3a-2150-47b0-b6e4-437e77a20172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldj2w" Jul 28 01:15:45.042984 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.042933 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/92b25e3a-2150-47b0-b6e4-437e77a20172-etc-selinux\") pod \"aws-ebs-csi-driver-node-ldj2w\" (UID: \"92b25e3a-2150-47b0-b6e4-437e77a20172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldj2w" Jul 28 01:15:45.044635 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.043033 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-multus-socket-dir-parent\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:45.044635 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.043216 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-log-socket\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.044635 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.043242 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b6db0343-a5f9-42ad-9869-539525bd315a-etc-sysctl-d\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:45.044635 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.043321 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-run-openvswitch\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.044635 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.043388 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-cnibin\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:45.044635 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.043449 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-host-var-lib-kubelet\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:45.044635 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.043514 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-host-run-netns\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:45.044635 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.043555 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrt4t\" (UniqueName: \"kubernetes.io/projected/3308b659-80af-41d9-a040-f056900489b3-kube-api-access-rrt4t\") pod \"network-metrics-daemon-zwr5s\" (UID: \"3308b659-80af-41d9-a040-f056900489b3\") " pod="openshift-multus/network-metrics-daemon-zwr5s" Jul 28 01:15:45.044635 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.043592 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-run-ovn\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.044635 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.043622 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-host-cni-netd\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.044635 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.044124 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-host-cni-bin\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.044635 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.044176 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/92b25e3a-2150-47b0-b6e4-437e77a20172-registration-dir\") pod \"aws-ebs-csi-driver-node-ldj2w\" (UID: \"92b25e3a-2150-47b0-b6e4-437e77a20172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldj2w" Jul 28 01:15:45.044635 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.044207 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-run-ovn\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.044635 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.044207 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-host-var-lib-cni-bin\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:45.044635 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.044234 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b6db0343-a5f9-42ad-9869-539525bd315a-host\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:45.044635 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.044258 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-host-cni-netd\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.044635 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.044268 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a29b525f-58ac-47f8-b759-d81742c281f4-system-cni-dir\") pod \"multus-additional-cni-plugins-2trzf\" (UID: \"a29b525f-58ac-47f8-b759-d81742c281f4\") " pod="openshift-multus/multus-additional-cni-plugins-2trzf" Jul 28 01:15:45.044635 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.044300 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/92b25e3a-2150-47b0-b6e4-437e77a20172-registration-dir\") pod \"aws-ebs-csi-driver-node-ldj2w\" (UID: \"92b25e3a-2150-47b0-b6e4-437e77a20172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldj2w" Jul 28 01:15:45.045665 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.044302 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a29b525f-58ac-47f8-b759-d81742c281f4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2trzf\" (UID: \"a29b525f-58ac-47f8-b759-d81742c281f4\") " pod="openshift-multus/multus-additional-cni-plugins-2trzf" Jul 28 01:15:45.045665 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.044341 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb797\" (UniqueName: \"kubernetes.io/projected/a29b525f-58ac-47f8-b759-d81742c281f4-kube-api-access-mb797\") pod \"multus-additional-cni-plugins-2trzf\" (UID: \"a29b525f-58ac-47f8-b759-d81742c281f4\") " pod="openshift-multus/multus-additional-cni-plugins-2trzf" Jul 28 01:15:45.045665 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.044348 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b6db0343-a5f9-42ad-9869-539525bd315a-host\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:45.045665 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.044384 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-host-run-netns\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.045665 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.044439 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a29b525f-58ac-47f8-b759-d81742c281f4-system-cni-dir\") pod \"multus-additional-cni-plugins-2trzf\" (UID: \"a29b525f-58ac-47f8-b759-d81742c281f4\") " pod="openshift-multus/multus-additional-cni-plugins-2trzf" Jul 28 01:15:45.045665 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.044479 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbdjv\" (UniqueName: \"kubernetes.io/projected/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-kube-api-access-vbdjv\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.045665 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.044627 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-host-run-netns\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.045665 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.044913 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-ovnkube-config\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.045665 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.044985 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a29b525f-58ac-47f8-b759-d81742c281f4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2trzf\" (UID: \"a29b525f-58ac-47f8-b759-d81742c281f4\") " pod="openshift-multus/multus-additional-cni-plugins-2trzf" Jul 28 01:15:45.045665 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.045000 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-ovn-node-metrics-cert\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.045665 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.045076 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b6db0343-a5f9-42ad-9869-539525bd315a-etc-modprobe-d\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:45.045665 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.045135 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-host-var-lib-cni-multus\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:45.045665 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.045178 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-var-lib-openvswitch\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.045665 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.045213 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.045665 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.045244 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6db0343-a5f9-42ad-9869-539525bd315a-etc-kubernetes\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:45.045665 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.045267 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a29b525f-58ac-47f8-b759-d81742c281f4-os-release\") pod \"multus-additional-cni-plugins-2trzf\" (UID: \"a29b525f-58ac-47f8-b759-d81742c281f4\") " pod="openshift-multus/multus-additional-cni-plugins-2trzf" Jul 28 01:15:45.045665 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.045287 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-os-release\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:45.046539 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.045371 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a29b525f-58ac-47f8-b759-d81742c281f4-os-release\") pod \"multus-additional-cni-plugins-2trzf\" (UID: \"a29b525f-58ac-47f8-b759-d81742c281f4\") " pod="openshift-multus/multus-additional-cni-plugins-2trzf" Jul 28 01:15:45.046539 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.045412 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhxk4\" (UniqueName: \"kubernetes.io/projected/bcb4dce7-214c-4f64-b5af-87ca7b0d6170-kube-api-access-hhxk4\") pod \"network-check-target-6swc2\" (UID: \"bcb4dce7-214c-4f64-b5af-87ca7b0d6170\") " pod="openshift-network-diagnostics/network-check-target-6swc2" Jul 28 01:15:45.046539 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.045449 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a29b525f-58ac-47f8-b759-d81742c281f4-cnibin\") pod \"multus-additional-cni-plugins-2trzf\" (UID: \"a29b525f-58ac-47f8-b759-d81742c281f4\") " pod="openshift-multus/multus-additional-cni-plugins-2trzf" Jul 28 01:15:45.046539 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.045483 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-etc-openvswitch\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.046539 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.045511 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-node-log\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.046539 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.045608 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-node-log\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.046539 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.045654 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-os-release\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:45.046539 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.045692 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-var-lib-openvswitch\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.046539 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.045751 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b6db0343-a5f9-42ad-9869-539525bd315a-etc-modprobe-d\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:45.046539 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.045779 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/07541f83-addd-4996-9a31-74860f8e3ba3-host-var-lib-cni-multus\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:45.046539 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.045806 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.046539 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.045844 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6db0343-a5f9-42ad-9869-539525bd315a-etc-kubernetes\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:45.046539 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.045975 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a29b525f-58ac-47f8-b759-d81742c281f4-cnibin\") pod \"multus-additional-cni-plugins-2trzf\" (UID: \"a29b525f-58ac-47f8-b759-d81742c281f4\") " pod="openshift-multus/multus-additional-cni-plugins-2trzf" Jul 28 01:15:45.046539 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.046072 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-etc-openvswitch\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.047280 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.046662 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d018a93b-4db3-45cb-8e11-919a45a35718-konnectivity-ca\") pod \"konnectivity-agent-xtl8m\" (UID: \"d018a93b-4db3-45cb-8e11-919a45a35718\") " pod="kube-system/konnectivity-agent-xtl8m" Jul 28 01:15:45.048676 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.048135 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d018a93b-4db3-45cb-8e11-919a45a35718-agent-certs\") pod \"konnectivity-agent-xtl8m\" (UID: \"d018a93b-4db3-45cb-8e11-919a45a35718\") " pod="kube-system/konnectivity-agent-xtl8m" Jul 28 01:15:45.049439 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.049316 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9ln8\" (UniqueName: \"kubernetes.io/projected/92b25e3a-2150-47b0-b6e4-437e77a20172-kube-api-access-f9ln8\") pod \"aws-ebs-csi-driver-node-ldj2w\" (UID: \"92b25e3a-2150-47b0-b6e4-437e77a20172\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldj2w" Jul 28 01:15:45.049632 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.049457 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6db0343-a5f9-42ad-9869-539525bd315a-tmp\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:45.050554 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.050513 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b6db0343-a5f9-42ad-9869-539525bd315a-etc-tuned\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:45.053027 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:45.052759 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 01:15:45.053027 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:45.052782 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 01:15:45.053027 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:45.052797 2565 projected.go:196] Error preparing data for projected volume kube-api-access-hhxk4 for pod openshift-network-diagnostics/network-check-target-6swc2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 01:15:45.053027 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:45.052875 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcb4dce7-214c-4f64-b5af-87ca7b0d6170-kube-api-access-hhxk4 podName:bcb4dce7-214c-4f64-b5af-87ca7b0d6170 nodeName:}" failed. No retries permitted until 2026-07-28 01:15:45.552838626 +0000 UTC m=+2.987236922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hhxk4" (UniqueName: "kubernetes.io/projected/bcb4dce7-214c-4f64-b5af-87ca7b0d6170-kube-api-access-hhxk4") pod "network-check-target-6swc2" (UID: "bcb4dce7-214c-4f64-b5af-87ca7b0d6170") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 01:15:45.056787 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.056760 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbdjv\" (UniqueName: \"kubernetes.io/projected/6bbd71b1-0a1b-4137-a8f7-4caa1c79911e-kube-api-access-vbdjv\") pod \"ovnkube-node-kbl47\" (UID: \"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.057233 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.057213 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w46w\" (UniqueName: \"kubernetes.io/projected/07541f83-addd-4996-9a31-74860f8e3ba3-kube-api-access-6w46w\") pod \"multus-rcvxw\" (UID: \"07541f83-addd-4996-9a31-74860f8e3ba3\") " pod="openshift-multus/multus-rcvxw" Jul 28 01:15:45.059125 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.059102 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb797\" (UniqueName: \"kubernetes.io/projected/a29b525f-58ac-47f8-b759-d81742c281f4-kube-api-access-mb797\") pod \"multus-additional-cni-plugins-2trzf\" (UID: \"a29b525f-58ac-47f8-b759-d81742c281f4\") " pod="openshift-multus/multus-additional-cni-plugins-2trzf" Jul 28 01:15:45.060093 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.060068 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh9gl\" (UniqueName: \"kubernetes.io/projected/b6db0343-a5f9-42ad-9869-539525bd315a-kube-api-access-fh9gl\") pod \"tuned-xjxrs\" (UID: \"b6db0343-a5f9-42ad-9869-539525bd315a\") " pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:45.061344 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.061320 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ltvc\" (UniqueName: \"kubernetes.io/projected/f32e8846-0caa-4b52-a5fd-ae65659fdea1-kube-api-access-7ltvc\") pod \"node-ca-zlwvl\" (UID: \"f32e8846-0caa-4b52-a5fd-ae65659fdea1\") " pod="openshift-image-registry/node-ca-zlwvl" Jul 28 01:15:45.100579 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.100531 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-23.ec2.internal" event={"ID":"3ff7e6d6dc372649b853c14fc595bb8f","Type":"ContainerStarted","Data":"985ee986b55cac745d1017c6b1dfe995285f506c26a8d20b1383ef1677231403"} Jul 28 01:15:45.101870 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.101842 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-23.ec2.internal" event={"ID":"6a601e81bd2ad6ddab885bb95322a480","Type":"ContainerStarted","Data":"a95487d7fab12d8cef29b993cc6d528e0969be6248a0a9e57572dc0ffa7b3bd5"} Jul 28 01:15:45.145826 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.145797 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3308b659-80af-41d9-a040-f056900489b3-metrics-certs\") pod \"network-metrics-daemon-zwr5s\" (UID: \"3308b659-80af-41d9-a040-f056900489b3\") " pod="openshift-multus/network-metrics-daemon-zwr5s" Jul 28 01:15:45.145956 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.145828 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a97a9823-8b73-453c-9d16-8907cc6b5116-hosts-file\") pod \"node-resolver-snwqc\" (UID: \"a97a9823-8b73-453c-9d16-8907cc6b5116\") " pod="openshift-dns/node-resolver-snwqc" Jul 28 01:15:45.145956 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.145873 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfz76\" (UniqueName: \"kubernetes.io/projected/ac3e75a4-c6a5-4b7f-a767-08bcc895d5af-kube-api-access-xfz76\") pod \"iptables-alerter-sf4nf\" (UID: \"ac3e75a4-c6a5-4b7f-a767-08bcc895d5af\") " pod="openshift-network-operator/iptables-alerter-sf4nf" Jul 28 01:15:45.145956 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.145901 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ac3e75a4-c6a5-4b7f-a767-08bcc895d5af-iptables-alerter-script\") pod \"iptables-alerter-sf4nf\" (UID: \"ac3e75a4-c6a5-4b7f-a767-08bcc895d5af\") " pod="openshift-network-operator/iptables-alerter-sf4nf" Jul 28 01:15:45.145956 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.145923 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a97a9823-8b73-453c-9d16-8907cc6b5116-tmp-dir\") pod \"node-resolver-snwqc\" (UID: \"a97a9823-8b73-453c-9d16-8907cc6b5116\") " pod="openshift-dns/node-resolver-snwqc" Jul 28 01:15:45.145956 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:45.145939 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 01:15:45.145956 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.145947 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5ppq\" (UniqueName: \"kubernetes.io/projected/a97a9823-8b73-453c-9d16-8907cc6b5116-kube-api-access-p5ppq\") pod \"node-resolver-snwqc\" (UID: \"a97a9823-8b73-453c-9d16-8907cc6b5116\") " pod="openshift-dns/node-resolver-snwqc" Jul 28 01:15:45.146256 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:45.146003 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3308b659-80af-41d9-a040-f056900489b3-metrics-certs podName:3308b659-80af-41d9-a040-f056900489b3 nodeName:}" failed. No retries permitted until 2026-07-28 01:15:45.645984785 +0000 UTC m=+3.080383094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3308b659-80af-41d9-a040-f056900489b3-metrics-certs") pod "network-metrics-daemon-zwr5s" (UID: "3308b659-80af-41d9-a040-f056900489b3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 01:15:45.146256 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.146023 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac3e75a4-c6a5-4b7f-a767-08bcc895d5af-host-slash\") pod \"iptables-alerter-sf4nf\" (UID: \"ac3e75a4-c6a5-4b7f-a767-08bcc895d5af\") " pod="openshift-network-operator/iptables-alerter-sf4nf" Jul 28 01:15:45.146256 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.146077 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/12e2c22a-bb31-4f97-869a-3c89eb64f837-kubelet-config\") pod \"global-pull-secret-syncer-8sv4q\" (UID: \"12e2c22a-bb31-4f97-869a-3c89eb64f837\") " pod="kube-system/global-pull-secret-syncer-8sv4q" Jul 28 01:15:45.146256 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.146109 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12e2c22a-bb31-4f97-869a-3c89eb64f837-original-pull-secret\") pod \"global-pull-secret-syncer-8sv4q\" (UID: \"12e2c22a-bb31-4f97-869a-3c89eb64f837\") " pod="kube-system/global-pull-secret-syncer-8sv4q" Jul 28 01:15:45.146256 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.146145 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac3e75a4-c6a5-4b7f-a767-08bcc895d5af-host-slash\") pod \"iptables-alerter-sf4nf\" (UID: \"ac3e75a4-c6a5-4b7f-a767-08bcc895d5af\") " pod="openshift-network-operator/iptables-alerter-sf4nf" Jul 28 01:15:45.146256 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.146139 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/12e2c22a-bb31-4f97-869a-3c89eb64f837-dbus\") pod \"global-pull-secret-syncer-8sv4q\" (UID: \"12e2c22a-bb31-4f97-869a-3c89eb64f837\") " pod="kube-system/global-pull-secret-syncer-8sv4q" Jul 28 01:15:45.146256 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.146221 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/12e2c22a-bb31-4f97-869a-3c89eb64f837-dbus\") pod \"global-pull-secret-syncer-8sv4q\" (UID: \"12e2c22a-bb31-4f97-869a-3c89eb64f837\") " pod="kube-system/global-pull-secret-syncer-8sv4q" Jul 28 01:15:45.146256 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.146223 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrt4t\" (UniqueName: \"kubernetes.io/projected/3308b659-80af-41d9-a040-f056900489b3-kube-api-access-rrt4t\") pod \"network-metrics-daemon-zwr5s\" (UID: \"3308b659-80af-41d9-a040-f056900489b3\") " pod="openshift-multus/network-metrics-daemon-zwr5s" Jul 28 01:15:45.146568 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.146278 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/12e2c22a-bb31-4f97-869a-3c89eb64f837-kubelet-config\") pod \"global-pull-secret-syncer-8sv4q\" (UID: \"12e2c22a-bb31-4f97-869a-3c89eb64f837\") " pod="kube-system/global-pull-secret-syncer-8sv4q" Jul 28 01:15:45.146568 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.146298 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a97a9823-8b73-453c-9d16-8907cc6b5116-tmp-dir\") pod \"node-resolver-snwqc\" (UID: \"a97a9823-8b73-453c-9d16-8907cc6b5116\") " pod="openshift-dns/node-resolver-snwqc" Jul 28 01:15:45.146568 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:45.146337 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 01:15:45.146568 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:45.146376 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12e2c22a-bb31-4f97-869a-3c89eb64f837-original-pull-secret podName:12e2c22a-bb31-4f97-869a-3c89eb64f837 nodeName:}" failed. No retries permitted until 2026-07-28 01:15:45.646363782 +0000 UTC m=+3.080762075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/12e2c22a-bb31-4f97-869a-3c89eb64f837-original-pull-secret") pod "global-pull-secret-syncer-8sv4q" (UID: "12e2c22a-bb31-4f97-869a-3c89eb64f837") : object "kube-system"/"original-pull-secret" not registered Jul 28 01:15:45.146568 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.146501 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a97a9823-8b73-453c-9d16-8907cc6b5116-hosts-file\") pod \"node-resolver-snwqc\" (UID: \"a97a9823-8b73-453c-9d16-8907cc6b5116\") " pod="openshift-dns/node-resolver-snwqc" Jul 28 01:15:45.146945 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.146921 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ac3e75a4-c6a5-4b7f-a767-08bcc895d5af-iptables-alerter-script\") pod \"iptables-alerter-sf4nf\" (UID: \"ac3e75a4-c6a5-4b7f-a767-08bcc895d5af\") " pod="openshift-network-operator/iptables-alerter-sf4nf" Jul 28 01:15:45.155411 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.155356 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfz76\" (UniqueName: \"kubernetes.io/projected/ac3e75a4-c6a5-4b7f-a767-08bcc895d5af-kube-api-access-xfz76\") pod \"iptables-alerter-sf4nf\" (UID: \"ac3e75a4-c6a5-4b7f-a767-08bcc895d5af\") " pod="openshift-network-operator/iptables-alerter-sf4nf" Jul 28 01:15:45.155775 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.155756 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5ppq\" (UniqueName: \"kubernetes.io/projected/a97a9823-8b73-453c-9d16-8907cc6b5116-kube-api-access-p5ppq\") pod \"node-resolver-snwqc\" (UID: \"a97a9823-8b73-453c-9d16-8907cc6b5116\") " pod="openshift-dns/node-resolver-snwqc" Jul 28 01:15:45.155854 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.155785 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrt4t\" (UniqueName: \"kubernetes.io/projected/3308b659-80af-41d9-a040-f056900489b3-kube-api-access-rrt4t\") pod \"network-metrics-daemon-zwr5s\" (UID: \"3308b659-80af-41d9-a040-f056900489b3\") " pod="openshift-multus/network-metrics-daemon-zwr5s" Jul 28 01:15:45.231850 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.231818 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rcvxw" Jul 28 01:15:45.241186 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:45.241152 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07541f83_addd_4996_9a31_74860f8e3ba3.slice/crio-7bc09b608d2edc3a55cc47a90b0125a0fc1d1a945c2495f2248fc8ff0b5b9000 WatchSource:0}: Error finding container 7bc09b608d2edc3a55cc47a90b0125a0fc1d1a945c2495f2248fc8ff0b5b9000: Status 404 returned error can't find the container with id 7bc09b608d2edc3a55cc47a90b0125a0fc1d1a945c2495f2248fc8ff0b5b9000 Jul 28 01:15:45.241298 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.241229 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zlwvl" Jul 28 01:15:45.249560 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:45.249538 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32e8846_0caa_4b52_a5fd_ae65659fdea1.slice/crio-29d12a2340708d0664860906d5bcbab627325835de241f9080d5d56c664aa440 WatchSource:0}: Error finding container 29d12a2340708d0664860906d5bcbab627325835de241f9080d5d56c664aa440: Status 404 returned error can't find the container with id 29d12a2340708d0664860906d5bcbab627325835de241f9080d5d56c664aa440 Jul 28 01:15:45.253347 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.253278 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:15:45.257697 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.257673 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xtl8m" Jul 28 01:15:45.262964 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:45.262931 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bbd71b1_0a1b_4137_a8f7_4caa1c79911e.slice/crio-05c2d8efe46c6f2cc69d2caaca5030851c5b513519a5a6d6afadc66cea1686fa WatchSource:0}: Error finding container 05c2d8efe46c6f2cc69d2caaca5030851c5b513519a5a6d6afadc66cea1686fa: Status 404 returned error can't find the container with id 05c2d8efe46c6f2cc69d2caaca5030851c5b513519a5a6d6afadc66cea1686fa Jul 28 01:15:45.266784 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.266762 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldj2w" Jul 28 01:15:45.277324 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.277302 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" Jul 28 01:15:45.285347 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.285325 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2trzf" Jul 28 01:15:45.294323 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.294298 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sf4nf" Jul 28 01:15:45.300197 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.300178 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-snwqc" Jul 28 01:15:45.650629 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.650590 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12e2c22a-bb31-4f97-869a-3c89eb64f837-original-pull-secret\") pod \"global-pull-secret-syncer-8sv4q\" (UID: \"12e2c22a-bb31-4f97-869a-3c89eb64f837\") " pod="kube-system/global-pull-secret-syncer-8sv4q" Jul 28 01:15:45.650807 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.650663 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhxk4\" (UniqueName: \"kubernetes.io/projected/bcb4dce7-214c-4f64-b5af-87ca7b0d6170-kube-api-access-hhxk4\") pod \"network-check-target-6swc2\" (UID: \"bcb4dce7-214c-4f64-b5af-87ca7b0d6170\") " pod="openshift-network-diagnostics/network-check-target-6swc2" Jul 28 01:15:45.650807 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.650704 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3308b659-80af-41d9-a040-f056900489b3-metrics-certs\") pod \"network-metrics-daemon-zwr5s\" (UID: \"3308b659-80af-41d9-a040-f056900489b3\") " pod="openshift-multus/network-metrics-daemon-zwr5s" Jul 28 01:15:45.650807 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:45.650712 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 01:15:45.650807 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:45.650787 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12e2c22a-bb31-4f97-869a-3c89eb64f837-original-pull-secret podName:12e2c22a-bb31-4f97-869a-3c89eb64f837 nodeName:}" failed. No retries permitted until 2026-07-28 01:15:46.650765638 +0000 UTC m=+4.085163934 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/12e2c22a-bb31-4f97-869a-3c89eb64f837-original-pull-secret") pod "global-pull-secret-syncer-8sv4q" (UID: "12e2c22a-bb31-4f97-869a-3c89eb64f837") : object "kube-system"/"original-pull-secret" not registered Jul 28 01:15:45.651012 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:45.650816 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 01:15:45.651012 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:45.650836 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 01:15:45.651012 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:45.650857 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 01:15:45.651012 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:45.650871 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3308b659-80af-41d9-a040-f056900489b3-metrics-certs podName:3308b659-80af-41d9-a040-f056900489b3 nodeName:}" failed. No retries permitted until 2026-07-28 01:15:46.650853049 +0000 UTC m=+4.085251349 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3308b659-80af-41d9-a040-f056900489b3-metrics-certs") pod "network-metrics-daemon-zwr5s" (UID: "3308b659-80af-41d9-a040-f056900489b3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 01:15:45.651012 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:45.650871 2565 projected.go:196] Error preparing data for projected volume kube-api-access-hhxk4 for pod openshift-network-diagnostics/network-check-target-6swc2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 01:15:45.651012 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:45.650917 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcb4dce7-214c-4f64-b5af-87ca7b0d6170-kube-api-access-hhxk4 podName:bcb4dce7-214c-4f64-b5af-87ca7b0d6170 nodeName:}" failed. No retries permitted until 2026-07-28 01:15:46.650905812 +0000 UTC m=+4.085304113 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hhxk4" (UniqueName: "kubernetes.io/projected/bcb4dce7-214c-4f64-b5af-87ca7b0d6170-kube-api-access-hhxk4") pod "network-check-target-6swc2" (UID: "bcb4dce7-214c-4f64-b5af-87ca7b0d6170") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 01:15:45.956245 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:45.956211 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd018a93b_4db3_45cb_8e11_919a45a35718.slice/crio-8b603c9ebb78feebe8f5ed1e7ae7d95b32a73a79064412c51a7e3ce16dc0e420 WatchSource:0}: Error finding container 8b603c9ebb78feebe8f5ed1e7ae7d95b32a73a79064412c51a7e3ce16dc0e420: Status 404 returned error can't find the container with id 8b603c9ebb78feebe8f5ed1e7ae7d95b32a73a79064412c51a7e3ce16dc0e420 Jul 28 01:15:45.965256 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.965225 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 01:10:45 +0000 UTC" deadline="2028-04-10 02:25:08.493174388 +0000 UTC" Jul 28 01:15:45.965256 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:45.965253 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14929h9m22.527924105s" Jul 28 01:15:46.104406 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:46.104355 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xtl8m" event={"ID":"d018a93b-4db3-45cb-8e11-919a45a35718","Type":"ContainerStarted","Data":"8b603c9ebb78feebe8f5ed1e7ae7d95b32a73a79064412c51a7e3ce16dc0e420"} Jul 28 01:15:46.105972 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:46.105941 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" event={"ID":"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e","Type":"ContainerStarted","Data":"05c2d8efe46c6f2cc69d2caaca5030851c5b513519a5a6d6afadc66cea1686fa"} Jul 28 01:15:46.107455 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:46.107399 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zlwvl" event={"ID":"f32e8846-0caa-4b52-a5fd-ae65659fdea1","Type":"ContainerStarted","Data":"29d12a2340708d0664860906d5bcbab627325835de241f9080d5d56c664aa440"} Jul 28 01:15:46.108823 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:46.108797 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rcvxw" event={"ID":"07541f83-addd-4996-9a31-74860f8e3ba3","Type":"ContainerStarted","Data":"7bc09b608d2edc3a55cc47a90b0125a0fc1d1a945c2495f2248fc8ff0b5b9000"} Jul 28 01:15:46.415824 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:46.415789 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92b25e3a_2150_47b0_b6e4_437e77a20172.slice/crio-b068528c953eedc0fab7b038d2717218140a66464554aeb95cd5cd18f72c7ba2 WatchSource:0}: Error finding container b068528c953eedc0fab7b038d2717218140a66464554aeb95cd5cd18f72c7ba2: Status 404 returned error can't find the container with id b068528c953eedc0fab7b038d2717218140a66464554aeb95cd5cd18f72c7ba2 Jul 28 01:15:46.427331 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:46.427297 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29b525f_58ac_47f8_b759_d81742c281f4.slice/crio-40931c7921a224bb73dbfc1290afe679064c4fdb7ececda18c0f32280103de94 WatchSource:0}: Error finding container 40931c7921a224bb73dbfc1290afe679064c4fdb7ececda18c0f32280103de94: Status 404 returned error can't find the container with id 40931c7921a224bb73dbfc1290afe679064c4fdb7ececda18c0f32280103de94 Jul 28 01:15:46.429615 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:46.429588 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda97a9823_8b73_453c_9d16_8907cc6b5116.slice/crio-b3a0709213acf0093fd43afbcc92c2bf89c16cfcdcdf6a8be3cf0f8303f1539e WatchSource:0}: Error finding container b3a0709213acf0093fd43afbcc92c2bf89c16cfcdcdf6a8be3cf0f8303f1539e: Status 404 returned error can't find the container with id b3a0709213acf0093fd43afbcc92c2bf89c16cfcdcdf6a8be3cf0f8303f1539e Jul 28 01:15:46.431271 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:15:46.431106 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac3e75a4_c6a5_4b7f_a767_08bcc895d5af.slice/crio-799eb1856fa58978f402ff7db35d6b974ae5a176239def2477e49e2aa2675268 WatchSource:0}: Error finding container 799eb1856fa58978f402ff7db35d6b974ae5a176239def2477e49e2aa2675268: Status 404 returned error can't find the container with id 799eb1856fa58978f402ff7db35d6b974ae5a176239def2477e49e2aa2675268 Jul 28 01:15:46.661402 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:46.661126 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12e2c22a-bb31-4f97-869a-3c89eb64f837-original-pull-secret\") pod \"global-pull-secret-syncer-8sv4q\" (UID: \"12e2c22a-bb31-4f97-869a-3c89eb64f837\") " pod="kube-system/global-pull-secret-syncer-8sv4q" Jul 28 01:15:46.661402 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:46.661341 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhxk4\" (UniqueName: \"kubernetes.io/projected/bcb4dce7-214c-4f64-b5af-87ca7b0d6170-kube-api-access-hhxk4\") pod \"network-check-target-6swc2\" (UID: \"bcb4dce7-214c-4f64-b5af-87ca7b0d6170\") " pod="openshift-network-diagnostics/network-check-target-6swc2" Jul 28 01:15:46.661402 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:46.661367 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3308b659-80af-41d9-a040-f056900489b3-metrics-certs\") pod \"network-metrics-daemon-zwr5s\" (UID: \"3308b659-80af-41d9-a040-f056900489b3\") " pod="openshift-multus/network-metrics-daemon-zwr5s" Jul 28 01:15:46.661670 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:46.661480 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 01:15:46.661670 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:46.661532 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3308b659-80af-41d9-a040-f056900489b3-metrics-certs podName:3308b659-80af-41d9-a040-f056900489b3 nodeName:}" failed. No retries permitted until 2026-07-28 01:15:48.661516878 +0000 UTC m=+6.095915172 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3308b659-80af-41d9-a040-f056900489b3-metrics-certs") pod "network-metrics-daemon-zwr5s" (UID: "3308b659-80af-41d9-a040-f056900489b3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 01:15:46.661910 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:46.661886 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 01:15:46.662000 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:46.661941 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12e2c22a-bb31-4f97-869a-3c89eb64f837-original-pull-secret podName:12e2c22a-bb31-4f97-869a-3c89eb64f837 nodeName:}" failed. No retries permitted until 2026-07-28 01:15:48.661928737 +0000 UTC m=+6.096327031 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/12e2c22a-bb31-4f97-869a-3c89eb64f837-original-pull-secret") pod "global-pull-secret-syncer-8sv4q" (UID: "12e2c22a-bb31-4f97-869a-3c89eb64f837") : object "kube-system"/"original-pull-secret" not registered Jul 28 01:15:46.662083 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:46.662000 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 01:15:46.662083 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:46.662013 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 01:15:46.662083 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:46.662022 2565 projected.go:196] Error preparing data for projected volume kube-api-access-hhxk4 for pod openshift-network-diagnostics/network-check-target-6swc2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 01:15:46.662083 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:46.662049 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcb4dce7-214c-4f64-b5af-87ca7b0d6170-kube-api-access-hhxk4 podName:bcb4dce7-214c-4f64-b5af-87ca7b0d6170 nodeName:}" failed. No retries permitted until 2026-07-28 01:15:48.662039561 +0000 UTC m=+6.096437854 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hhxk4" (UniqueName: "kubernetes.io/projected/bcb4dce7-214c-4f64-b5af-87ca7b0d6170-kube-api-access-hhxk4") pod "network-check-target-6swc2" (UID: "bcb4dce7-214c-4f64-b5af-87ca7b0d6170") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 01:15:46.965436 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:46.965370 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 01:10:45 +0000 UTC" deadline="2028-03-30 07:57:41.738539165 +0000 UTC" Jul 28 01:15:46.965436 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:46.965435 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14670h41m54.773107993s" Jul 28 01:15:47.099375 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:47.098471 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6swc2" Jul 28 01:15:47.099375 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:47.098602 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6swc2" podUID="bcb4dce7-214c-4f64-b5af-87ca7b0d6170" Jul 28 01:15:47.099375 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:47.099044 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv4q" Jul 28 01:15:47.099375 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:47.099161 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8sv4q" podUID="12e2c22a-bb31-4f97-869a-3c89eb64f837" Jul 28 01:15:47.099375 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:47.099239 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwr5s" Jul 28 01:15:47.099375 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:47.099325 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwr5s" podUID="3308b659-80af-41d9-a040-f056900489b3" Jul 28 01:15:47.116634 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:47.116597 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2trzf" event={"ID":"a29b525f-58ac-47f8-b759-d81742c281f4","Type":"ContainerStarted","Data":"40931c7921a224bb73dbfc1290afe679064c4fdb7ececda18c0f32280103de94"} Jul 28 01:15:47.119019 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:47.118986 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-snwqc" event={"ID":"a97a9823-8b73-453c-9d16-8907cc6b5116","Type":"ContainerStarted","Data":"b3a0709213acf0093fd43afbcc92c2bf89c16cfcdcdf6a8be3cf0f8303f1539e"} Jul 28 01:15:47.121248 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:47.121215 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sf4nf" event={"ID":"ac3e75a4-c6a5-4b7f-a767-08bcc895d5af","Type":"ContainerStarted","Data":"799eb1856fa58978f402ff7db35d6b974ae5a176239def2477e49e2aa2675268"} Jul 28 01:15:47.130664 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:47.130581 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" event={"ID":"b6db0343-a5f9-42ad-9869-539525bd315a","Type":"ContainerStarted","Data":"c8d5c13802c83dab6b92e241636280cbe89afd0ed720df86a9dbbf497558c352"} Jul 28 01:15:47.137286 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:47.137209 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldj2w" event={"ID":"92b25e3a-2150-47b0-b6e4-437e77a20172","Type":"ContainerStarted","Data":"b068528c953eedc0fab7b038d2717218140a66464554aeb95cd5cd18f72c7ba2"} Jul 28 01:15:47.148241 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:47.148204 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-23.ec2.internal" event={"ID":"3ff7e6d6dc372649b853c14fc595bb8f","Type":"ContainerStarted","Data":"d277eaf6e62a6ffce434cb2abc74d8323ef0fc7ddaabb6e86236856daff421f1"} Jul 28 01:15:47.161183 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:47.161153 2565 generic.go:356] "Generic (PLEG): container finished" podID="6a601e81bd2ad6ddab885bb95322a480" containerID="1d38deae9e107accf65ed625fe981d67ed789216ecb74a3703b4eda9191547e3" exitCode=0 Jul 28 01:15:47.161309 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:47.161199 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-23.ec2.internal" event={"ID":"6a601e81bd2ad6ddab885bb95322a480","Type":"ContainerDied","Data":"1d38deae9e107accf65ed625fe981d67ed789216ecb74a3703b4eda9191547e3"} Jul 28 01:15:47.180706 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:47.180643 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-23.ec2.internal" podStartSLOduration=3.180626375 podStartE2EDuration="3.180626375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:15:47.166197151 +0000 UTC m=+4.600595470" watchObservedRunningTime="2026-07-28 01:15:47.180626375 +0000 UTC m=+4.615024695" Jul 28 01:15:48.680382 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:48.680341 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhxk4\" (UniqueName: \"kubernetes.io/projected/bcb4dce7-214c-4f64-b5af-87ca7b0d6170-kube-api-access-hhxk4\") pod \"network-check-target-6swc2\" (UID: \"bcb4dce7-214c-4f64-b5af-87ca7b0d6170\") " pod="openshift-network-diagnostics/network-check-target-6swc2" Jul 28 01:15:48.680878 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:48.680404 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3308b659-80af-41d9-a040-f056900489b3-metrics-certs\") pod \"network-metrics-daemon-zwr5s\" (UID: \"3308b659-80af-41d9-a040-f056900489b3\") " pod="openshift-multus/network-metrics-daemon-zwr5s" Jul 28 01:15:48.680878 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:48.680458 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12e2c22a-bb31-4f97-869a-3c89eb64f837-original-pull-secret\") pod \"global-pull-secret-syncer-8sv4q\" (UID: \"12e2c22a-bb31-4f97-869a-3c89eb64f837\") " pod="kube-system/global-pull-secret-syncer-8sv4q" Jul 28 01:15:48.680878 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:48.680601 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 01:15:48.680878 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:48.680667 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12e2c22a-bb31-4f97-869a-3c89eb64f837-original-pull-secret podName:12e2c22a-bb31-4f97-869a-3c89eb64f837 nodeName:}" failed. No retries permitted until 2026-07-28 01:15:52.680648189 +0000 UTC m=+10.115046498 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/12e2c22a-bb31-4f97-869a-3c89eb64f837-original-pull-secret") pod "global-pull-secret-syncer-8sv4q" (UID: "12e2c22a-bb31-4f97-869a-3c89eb64f837") : object "kube-system"/"original-pull-secret" not registered Jul 28 01:15:48.681153 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:48.681092 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 01:15:48.681153 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:48.681143 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 01:15:48.681275 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:48.681161 2565 projected.go:196] Error preparing data for projected volume kube-api-access-hhxk4 for pod openshift-network-diagnostics/network-check-target-6swc2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 01:15:48.681275 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:48.681210 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcb4dce7-214c-4f64-b5af-87ca7b0d6170-kube-api-access-hhxk4 podName:bcb4dce7-214c-4f64-b5af-87ca7b0d6170 nodeName:}" failed. No retries permitted until 2026-07-28 01:15:52.681192383 +0000 UTC m=+10.115590693 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hhxk4" (UniqueName: "kubernetes.io/projected/bcb4dce7-214c-4f64-b5af-87ca7b0d6170-kube-api-access-hhxk4") pod "network-check-target-6swc2" (UID: "bcb4dce7-214c-4f64-b5af-87ca7b0d6170") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 01:15:48.681389 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:48.681281 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 01:15:48.681389 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:48.681316 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3308b659-80af-41d9-a040-f056900489b3-metrics-certs podName:3308b659-80af-41d9-a040-f056900489b3 nodeName:}" failed. No retries permitted until 2026-07-28 01:15:52.681305179 +0000 UTC m=+10.115703489 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3308b659-80af-41d9-a040-f056900489b3-metrics-certs") pod "network-metrics-daemon-zwr5s" (UID: "3308b659-80af-41d9-a040-f056900489b3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 01:15:49.096242 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:49.096206 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv4q" Jul 28 01:15:49.096423 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:49.096360 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8sv4q" podUID="12e2c22a-bb31-4f97-869a-3c89eb64f837" Jul 28 01:15:49.096784 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:49.096763 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwr5s" Jul 28 01:15:49.096918 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:49.096897 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwr5s" podUID="3308b659-80af-41d9-a040-f056900489b3" Jul 28 01:15:49.097137 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:49.097063 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6swc2" Jul 28 01:15:49.097242 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:49.097151 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6swc2" podUID="bcb4dce7-214c-4f64-b5af-87ca7b0d6170" Jul 28 01:15:51.095953 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:51.095437 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwr5s" Jul 28 01:15:51.095953 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:51.095437 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv4q" Jul 28 01:15:51.095953 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:51.095585 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6swc2" Jul 28 01:15:51.095953 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:51.095586 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwr5s" podUID="3308b659-80af-41d9-a040-f056900489b3" Jul 28 01:15:51.095953 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:51.095661 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8sv4q" podUID="12e2c22a-bb31-4f97-869a-3c89eb64f837" Jul 28 01:15:51.095953 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:51.095729 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6swc2" podUID="bcb4dce7-214c-4f64-b5af-87ca7b0d6170" Jul 28 01:15:52.717878 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:52.717843 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhxk4\" (UniqueName: \"kubernetes.io/projected/bcb4dce7-214c-4f64-b5af-87ca7b0d6170-kube-api-access-hhxk4\") pod \"network-check-target-6swc2\" (UID: \"bcb4dce7-214c-4f64-b5af-87ca7b0d6170\") " pod="openshift-network-diagnostics/network-check-target-6swc2" Jul 28 01:15:52.718382 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:52.717895 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3308b659-80af-41d9-a040-f056900489b3-metrics-certs\") pod \"network-metrics-daemon-zwr5s\" (UID: \"3308b659-80af-41d9-a040-f056900489b3\") " pod="openshift-multus/network-metrics-daemon-zwr5s" Jul 28 01:15:52.718382 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:52.717942 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12e2c22a-bb31-4f97-869a-3c89eb64f837-original-pull-secret\") pod \"global-pull-secret-syncer-8sv4q\" (UID: \"12e2c22a-bb31-4f97-869a-3c89eb64f837\") " pod="kube-system/global-pull-secret-syncer-8sv4q" Jul 28 01:15:52.718382 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:52.718078 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 01:15:52.718382 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:52.718135 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12e2c22a-bb31-4f97-869a-3c89eb64f837-original-pull-secret podName:12e2c22a-bb31-4f97-869a-3c89eb64f837 nodeName:}" failed. No retries permitted until 2026-07-28 01:16:00.718118275 +0000 UTC m=+18.152516590 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/12e2c22a-bb31-4f97-869a-3c89eb64f837-original-pull-secret") pod "global-pull-secret-syncer-8sv4q" (UID: "12e2c22a-bb31-4f97-869a-3c89eb64f837") : object "kube-system"/"original-pull-secret" not registered Jul 28 01:15:52.718382 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:52.718218 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 01:15:52.718382 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:52.718236 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 01:15:52.718382 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:52.718247 2565 projected.go:196] Error preparing data for projected volume kube-api-access-hhxk4 for pod openshift-network-diagnostics/network-check-target-6swc2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 01:15:52.718382 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:52.718280 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcb4dce7-214c-4f64-b5af-87ca7b0d6170-kube-api-access-hhxk4 podName:bcb4dce7-214c-4f64-b5af-87ca7b0d6170 nodeName:}" failed. No retries permitted until 2026-07-28 01:16:00.718268982 +0000 UTC m=+18.152667290 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hhxk4" (UniqueName: "kubernetes.io/projected/bcb4dce7-214c-4f64-b5af-87ca7b0d6170-kube-api-access-hhxk4") pod "network-check-target-6swc2" (UID: "bcb4dce7-214c-4f64-b5af-87ca7b0d6170") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 01:15:52.718382 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:52.718334 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 01:15:52.718382 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:52.718364 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3308b659-80af-41d9-a040-f056900489b3-metrics-certs podName:3308b659-80af-41d9-a040-f056900489b3 nodeName:}" failed. No retries permitted until 2026-07-28 01:16:00.718352543 +0000 UTC m=+18.152750836 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3308b659-80af-41d9-a040-f056900489b3-metrics-certs") pod "network-metrics-daemon-zwr5s" (UID: "3308b659-80af-41d9-a040-f056900489b3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 01:15:53.096588 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:53.096375 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv4q" Jul 28 01:15:53.096588 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:53.096496 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8sv4q" podUID="12e2c22a-bb31-4f97-869a-3c89eb64f837" Jul 28 01:15:53.096588 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:53.096552 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwr5s" Jul 28 01:15:53.096868 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:53.096683 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwr5s" podUID="3308b659-80af-41d9-a040-f056900489b3" Jul 28 01:15:53.096868 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:53.096731 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6swc2" Jul 28 01:15:53.096868 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:53.096806 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6swc2" podUID="bcb4dce7-214c-4f64-b5af-87ca7b0d6170" Jul 28 01:15:55.095614 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:55.095577 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv4q" Jul 28 01:15:55.096147 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:55.095710 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8sv4q" podUID="12e2c22a-bb31-4f97-869a-3c89eb64f837" Jul 28 01:15:55.096147 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:55.095768 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6swc2" Jul 28 01:15:55.096147 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:55.095842 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6swc2" podUID="bcb4dce7-214c-4f64-b5af-87ca7b0d6170" Jul 28 01:15:55.096147 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:55.095891 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwr5s" Jul 28 01:15:55.096147 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:55.095957 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwr5s" podUID="3308b659-80af-41d9-a040-f056900489b3" Jul 28 01:15:57.095517 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:57.095481 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv4q" Jul 28 01:15:57.095517 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:57.095500 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwr5s" Jul 28 01:15:57.096045 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:57.095601 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8sv4q" podUID="12e2c22a-bb31-4f97-869a-3c89eb64f837" Jul 28 01:15:57.096045 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:57.095675 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwr5s" podUID="3308b659-80af-41d9-a040-f056900489b3" Jul 28 01:15:57.096045 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:57.095706 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6swc2" Jul 28 01:15:57.096045 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:57.095801 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6swc2" podUID="bcb4dce7-214c-4f64-b5af-87ca7b0d6170" Jul 28 01:15:59.096184 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:59.096145 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv4q" Jul 28 01:15:59.096641 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:59.096145 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6swc2" Jul 28 01:15:59.096641 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:15:59.096153 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwr5s" Jul 28 01:15:59.096641 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:59.096316 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8sv4q" podUID="12e2c22a-bb31-4f97-869a-3c89eb64f837" Jul 28 01:15:59.096641 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:59.096429 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwr5s" podUID="3308b659-80af-41d9-a040-f056900489b3" Jul 28 01:15:59.096641 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:15:59.096516 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6swc2" podUID="bcb4dce7-214c-4f64-b5af-87ca7b0d6170" Jul 28 01:16:00.782736 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:00.782701 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhxk4\" (UniqueName: \"kubernetes.io/projected/bcb4dce7-214c-4f64-b5af-87ca7b0d6170-kube-api-access-hhxk4\") pod \"network-check-target-6swc2\" (UID: \"bcb4dce7-214c-4f64-b5af-87ca7b0d6170\") " pod="openshift-network-diagnostics/network-check-target-6swc2" Jul 28 01:16:00.783334 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:00.782747 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3308b659-80af-41d9-a040-f056900489b3-metrics-certs\") pod \"network-metrics-daemon-zwr5s\" (UID: \"3308b659-80af-41d9-a040-f056900489b3\") " pod="openshift-multus/network-metrics-daemon-zwr5s" Jul 28 01:16:00.783334 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:00.782796 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12e2c22a-bb31-4f97-869a-3c89eb64f837-original-pull-secret\") pod \"global-pull-secret-syncer-8sv4q\" (UID: \"12e2c22a-bb31-4f97-869a-3c89eb64f837\") " pod="kube-system/global-pull-secret-syncer-8sv4q" Jul 28 01:16:00.783334 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:00.782889 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 01:16:00.783334 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:00.782913 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 01:16:00.783334 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:00.782963 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12e2c22a-bb31-4f97-869a-3c89eb64f837-original-pull-secret podName:12e2c22a-bb31-4f97-869a-3c89eb64f837 nodeName:}" failed. No retries permitted until 2026-07-28 01:16:16.782947385 +0000 UTC m=+34.217345677 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/12e2c22a-bb31-4f97-869a-3c89eb64f837-original-pull-secret") pod "global-pull-secret-syncer-8sv4q" (UID: "12e2c22a-bb31-4f97-869a-3c89eb64f837") : object "kube-system"/"original-pull-secret" not registered Jul 28 01:16:00.783334 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:00.782918 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 01:16:00.783334 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:00.783004 2565 projected.go:196] Error preparing data for projected volume kube-api-access-hhxk4 for pod openshift-network-diagnostics/network-check-target-6swc2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 01:16:00.783334 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:00.782888 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 01:16:00.783334 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:00.783076 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcb4dce7-214c-4f64-b5af-87ca7b0d6170-kube-api-access-hhxk4 podName:bcb4dce7-214c-4f64-b5af-87ca7b0d6170 nodeName:}" failed. No retries permitted until 2026-07-28 01:16:16.783057155 +0000 UTC m=+34.217455467 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hhxk4" (UniqueName: "kubernetes.io/projected/bcb4dce7-214c-4f64-b5af-87ca7b0d6170-kube-api-access-hhxk4") pod "network-check-target-6swc2" (UID: "bcb4dce7-214c-4f64-b5af-87ca7b0d6170") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 01:16:00.783334 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:00.783104 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3308b659-80af-41d9-a040-f056900489b3-metrics-certs podName:3308b659-80af-41d9-a040-f056900489b3 nodeName:}" failed. No retries permitted until 2026-07-28 01:16:16.783089323 +0000 UTC m=+34.217487619 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3308b659-80af-41d9-a040-f056900489b3-metrics-certs") pod "network-metrics-daemon-zwr5s" (UID: "3308b659-80af-41d9-a040-f056900489b3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 01:16:01.096262 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:01.096168 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwr5s" Jul 28 01:16:01.096425 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:01.096168 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6swc2" Jul 28 01:16:01.096425 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:01.096321 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwr5s" podUID="3308b659-80af-41d9-a040-f056900489b3" Jul 28 01:16:01.096425 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:01.096168 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv4q" Jul 28 01:16:01.096425 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:01.096398 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6swc2" podUID="bcb4dce7-214c-4f64-b5af-87ca7b0d6170" Jul 28 01:16:01.096669 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:01.096498 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8sv4q" podUID="12e2c22a-bb31-4f97-869a-3c89eb64f837" Jul 28 01:16:03.096138 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:03.096102 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv4q" Jul 28 01:16:03.096569 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:03.096225 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwr5s" Jul 28 01:16:03.096569 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:03.096221 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8sv4q" podUID="12e2c22a-bb31-4f97-869a-3c89eb64f837" Jul 28 01:16:03.096569 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:03.096370 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6swc2" Jul 28 01:16:03.096569 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:03.096367 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwr5s" podUID="3308b659-80af-41d9-a040-f056900489b3" Jul 28 01:16:03.096569 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:03.096482 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6swc2" podUID="bcb4dce7-214c-4f64-b5af-87ca7b0d6170" Jul 28 01:16:05.096076 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:05.095771 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv4q" Jul 28 01:16:05.096907 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:05.095773 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwr5s" Jul 28 01:16:05.096907 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:05.096163 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8sv4q" podUID="12e2c22a-bb31-4f97-869a-3c89eb64f837" Jul 28 01:16:05.096907 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:05.095827 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6swc2" Jul 28 01:16:05.096907 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:05.096301 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwr5s" podUID="3308b659-80af-41d9-a040-f056900489b3" Jul 28 01:16:05.096907 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:05.096424 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6swc2" podUID="bcb4dce7-214c-4f64-b5af-87ca7b0d6170" Jul 28 01:16:05.207017 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:05.206964 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-23.ec2.internal" event={"ID":"6a601e81bd2ad6ddab885bb95322a480","Type":"ContainerStarted","Data":"13da3cc1fcf6305f6009b9514c227fd228a0ca1f46b970abbe555c5f0584faaf"} Jul 28 01:16:05.208342 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:05.208318 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zlwvl" event={"ID":"f32e8846-0caa-4b52-a5fd-ae65659fdea1","Type":"ContainerStarted","Data":"1a5bce8f320776c7993fa24117e206176b8261024fb4a00215ed2c5c0fe083bb"} Jul 28 01:16:05.209872 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:05.209836 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rcvxw" event={"ID":"07541f83-addd-4996-9a31-74860f8e3ba3","Type":"ContainerStarted","Data":"6091f40325b8b783d024daf908a9eff673463763fea97f7a205867ae8c117812"} Jul 28 01:16:05.211427 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:05.211405 2565 generic.go:356] "Generic (PLEG): container finished" podID="a29b525f-58ac-47f8-b759-d81742c281f4" containerID="4a34a4af365571b9567731e67de34e7345294de8870a2d310cdf436356fecba1" exitCode=0 Jul 28 01:16:05.211511 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:05.211470 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2trzf" event={"ID":"a29b525f-58ac-47f8-b759-d81742c281f4","Type":"ContainerDied","Data":"4a34a4af365571b9567731e67de34e7345294de8870a2d310cdf436356fecba1"} Jul 28 01:16:05.212933 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:05.212902 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-snwqc" event={"ID":"a97a9823-8b73-453c-9d16-8907cc6b5116","Type":"ContainerStarted","Data":"ee608a979c3a26ca5808f12ba83260874004d916f09d0928ad2266042cde468e"} Jul 28 01:16:05.214382 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:05.214354 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xtl8m" event={"ID":"d018a93b-4db3-45cb-8e11-919a45a35718","Type":"ContainerStarted","Data":"d5f6b3136a4c62169b6f9ca75c61c4b67e71d96d04923aeb9913aa9e49840172"} Jul 28 01:16:05.216403 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:05.216385 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" event={"ID":"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e","Type":"ContainerStarted","Data":"f20642e961bd5b0ee532d1a4285651859efc7aaced3959e22b0365812f30eb7f"} Jul 28 01:16:05.216497 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:05.216453 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" event={"ID":"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e","Type":"ContainerStarted","Data":"de59c69b66aafa6a0739b1a35965badcba3b7d922133a5d1b375e8263be2d07a"} Jul 28 01:16:05.217790 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:05.217772 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" event={"ID":"b6db0343-a5f9-42ad-9869-539525bd315a","Type":"ContainerStarted","Data":"a1b021a1ffdeb5207acd3ad4a661bab2151fd4d91164787da8a52e5c6556cf29"} Jul 28 01:16:05.219117 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:05.219099 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldj2w" event={"ID":"92b25e3a-2150-47b0-b6e4-437e77a20172","Type":"ContainerStarted","Data":"8d9f15e11b93234c367b0d5294704745ac3ce97825f9d23155a8234d590fc513"} Jul 28 01:16:05.221208 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:05.221173 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-23.ec2.internal" podStartSLOduration=21.221162828 podStartE2EDuration="21.221162828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:16:05.220999302 +0000 UTC m=+22.655397616" watchObservedRunningTime="2026-07-28 01:16:05.221162828 +0000 UTC m=+22.655561141" Jul 28 01:16:05.235150 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:05.235118 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zlwvl" podStartSLOduration=10.899193114 podStartE2EDuration="22.235106379s" totalImagesPullingTime="11.335913265s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:16:05.234945462 +0000 UTC m=+22.669343775" watchObservedRunningTime="2026-07-28 01:16:05.235106379 +0000 UTC m=+22.669504691" Jul 28 01:16:05.248416 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:05.248382 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xtl8m" podStartSLOduration=3.524285247 podStartE2EDuration="22.24837042s" totalImagesPullingTime="18.724085173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:16:05.247740601 +0000 UTC m=+22.682138917" watchObservedRunningTime="2026-07-28 01:16:05.24837042 +0000 UTC m=+22.682768734" Jul 28 01:16:05.261537 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:05.261504 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-snwqc" podStartSLOduration=4.011127903 podStartE2EDuration="22.261493585s" totalImagesPullingTime="18.250365682s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:16:05.261306056 +0000 UTC m=+22.695704375" watchObservedRunningTime="2026-07-28 01:16:05.261493585 +0000 UTC m=+22.695891898" Jul 28 01:16:05.278255 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:05.278224 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rcvxw" podStartSLOduration=2.826267766 podStartE2EDuration="22.278214089s" totalImagesPullingTime="19.451946323s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:16:05.277708669 +0000 UTC m=+22.712106981" watchObservedRunningTime="2026-07-28 01:16:05.278214089 +0000 UTC m=+22.712612428" Jul 28 01:16:05.294576 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:05.294537 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xjxrs" podStartSLOduration=4.033602689 podStartE2EDuration="22.294524561s" totalImagesPullingTime="18.260921872s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:16:05.29415055 +0000 UTC m=+22.728548859" watchObservedRunningTime="2026-07-28 01:16:05.294524561 +0000 UTC m=+22.728922873" Jul 28 01:16:06.225133 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:06.225100 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" event={"ID":"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e","Type":"ContainerStarted","Data":"0c117a1d998b9f08500ec309f90b4ffec2294120571a9f30ceee9e9e5e8db2ac"} Jul 28 01:16:06.225547 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:06.225231 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" event={"ID":"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e","Type":"ContainerStarted","Data":"3f018ebc0edd7221e7488ebf48c64a84252141ba61fabd7fcce312950c0e00e9"} Jul 28 01:16:06.225547 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:06.225282 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" event={"ID":"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e","Type":"ContainerStarted","Data":"173501d7e794b753d7c98e9615bbecf0b684a237b587d4f85f194c66719ddf04"} Jul 28 01:16:06.225547 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:06.225328 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" event={"ID":"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e","Type":"ContainerStarted","Data":"9793050833b0479a02c98acbe2d720cff899e171680a7fbafb75c607b643a2fd"} Jul 28 01:16:06.282774 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:06.282663 2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 28 01:16:06.997602 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:06.997497 2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T01:16:06.282679443Z","UUID":"809e85e5-2ec8-4d2f-bc6f-a7e20d014250","Handler":null,"Name":"","Endpoint":""} Jul 28 01:16:06.999219 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:06.999191 2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 28 01:16:06.999342 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:06.999240 2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 28 01:16:07.095258 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:07.095228 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv4q" Jul 28 01:16:07.095426 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:07.095228 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6swc2" Jul 28 01:16:07.095426 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:07.095347 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8sv4q" podUID="12e2c22a-bb31-4f97-869a-3c89eb64f837" Jul 28 01:16:07.095534 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:07.095434 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6swc2" podUID="bcb4dce7-214c-4f64-b5af-87ca7b0d6170" Jul 28 01:16:07.095534 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:07.095230 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwr5s" Jul 28 01:16:07.095621 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:07.095551 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwr5s" podUID="3308b659-80af-41d9-a040-f056900489b3" Jul 28 01:16:07.230607 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:07.230571 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sf4nf" event={"ID":"ac3e75a4-c6a5-4b7f-a767-08bcc895d5af","Type":"ContainerStarted","Data":"8ce2f7bd5f0da3737aab13f1d9b875dd9ba60885ad2869417257838908c7e1fa"} Jul 28 01:16:07.232490 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:07.232439 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldj2w" event={"ID":"92b25e3a-2150-47b0-b6e4-437e77a20172","Type":"ContainerStarted","Data":"b8dbd236ef69ddcbfbb8b7cc13f7861a406e41fd3e7b9731ff645f2824e6004e"} Jul 28 01:16:07.244162 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:07.244115 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-sf4nf" podStartSLOduration=5.994657141 podStartE2EDuration="24.244101753s" totalImagesPullingTime="18.249444612s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:16:07.243997193 +0000 UTC m=+24.678395507" watchObservedRunningTime="2026-07-28 01:16:07.244101753 +0000 UTC m=+24.678500065" Jul 28 01:16:07.880262 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:07.880162 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xtl8m" Jul 28 01:16:07.880781 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:07.880765 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xtl8m" Jul 28 01:16:08.237742 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:08.237706 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldj2w" event={"ID":"92b25e3a-2150-47b0-b6e4-437e77a20172","Type":"ContainerStarted","Data":"59934a516d1e5088b8724d53749ddefe0efb978ed90d436111b5b09b3cba155d"} Jul 28 01:16:08.238163 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:08.238137 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xtl8m" Jul 28 01:16:08.238498 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:08.238480 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xtl8m" Jul 28 01:16:08.258150 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:08.256286 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldj2w" podStartSLOduration=4.823700093 podStartE2EDuration="25.256259019s" totalImagesPullingTime="20.432558926s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:15:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:16:08.255583971 +0000 UTC m=+25.689982276" watchObservedRunningTime="2026-07-28 01:16:08.256259019 +0000 UTC m=+25.690657333" Jul 28 01:16:09.095573 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:09.095356 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv4q" Jul 28 01:16:09.095729 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:09.095356 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6swc2" Jul 28 01:16:09.095729 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:09.095679 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8sv4q" podUID="12e2c22a-bb31-4f97-869a-3c89eb64f837" Jul 28 01:16:09.095729 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:09.095356 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwr5s" Jul 28 01:16:09.095831 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:09.095736 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6swc2" podUID="bcb4dce7-214c-4f64-b5af-87ca7b0d6170" Jul 28 01:16:09.095831 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:09.095811 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwr5s" podUID="3308b659-80af-41d9-a040-f056900489b3" Jul 28 01:16:10.244477 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:10.244443 2565 generic.go:356] "Generic (PLEG): container finished" podID="a29b525f-58ac-47f8-b759-d81742c281f4" containerID="1e18a7af30125531d6f744761e2692ec67e95225938047a4ea1885ffe4c95865" exitCode=0 Jul 28 01:16:10.245058 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:10.244527 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2trzf" event={"ID":"a29b525f-58ac-47f8-b759-d81742c281f4","Type":"ContainerDied","Data":"1e18a7af30125531d6f744761e2692ec67e95225938047a4ea1885ffe4c95865"} Jul 28 01:16:10.248006 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:10.247986 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" event={"ID":"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e","Type":"ContainerStarted","Data":"91494196e439fbbe1c811ea7cce1c356663005d8dacd980bbeb180f6b6564df6"} Jul 28 01:16:11.095340 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:11.095305 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwr5s" Jul 28 01:16:11.095340 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:11.095326 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv4q" Jul 28 01:16:11.095340 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:11.095311 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6swc2" Jul 28 01:16:11.095561 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:11.095419 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8sv4q" podUID="12e2c22a-bb31-4f97-869a-3c89eb64f837" Jul 28 01:16:11.095561 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:11.095509 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6swc2" podUID="bcb4dce7-214c-4f64-b5af-87ca7b0d6170" Jul 28 01:16:11.095634 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:11.095606 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwr5s" podUID="3308b659-80af-41d9-a040-f056900489b3" Jul 28 01:16:12.255194 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:12.255011 2565 generic.go:356] "Generic (PLEG): container finished" podID="a29b525f-58ac-47f8-b759-d81742c281f4" containerID="4b41543146c087b689924e1a06b8d0875b2b16d7dbe80037f6750765c24e59a3" exitCode=0 Jul 28 01:16:12.255789 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:12.255091 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2trzf" event={"ID":"a29b525f-58ac-47f8-b759-d81742c281f4","Type":"ContainerDied","Data":"4b41543146c087b689924e1a06b8d0875b2b16d7dbe80037f6750765c24e59a3"} Jul 28 01:16:12.258887 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:12.258860 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" event={"ID":"6bbd71b1-0a1b-4137-a8f7-4caa1c79911e","Type":"ContainerStarted","Data":"63b8c34bfbbd5fdadb04e4e67bc77edc61b11c1fa7ca95a5d256522ec0e1176b"} Jul 28 01:16:12.259062 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:12.259050 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:16:12.259119 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:12.259086 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:16:12.274610 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:12.274590 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:16:12.299844 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:12.299800 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" podStartSLOduration=9.707150341 podStartE2EDuration="29.299784533s" totalImagesPullingTime="19.592634192s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:16:12.298681877 +0000 UTC m=+29.733080192" watchObservedRunningTime="2026-07-28 01:16:12.299784533 +0000 UTC m=+29.734182847" Jul 28 01:16:13.097104 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:13.097075 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6swc2" Jul 28 01:16:13.097305 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:13.097156 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwr5s" Jul 28 01:16:13.097305 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:13.097118 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv4q" Jul 28 01:16:13.097305 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:13.097263 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6swc2" podUID="bcb4dce7-214c-4f64-b5af-87ca7b0d6170" Jul 28 01:16:13.097458 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:13.097359 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8sv4q" podUID="12e2c22a-bb31-4f97-869a-3c89eb64f837" Jul 28 01:16:13.097505 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:13.097469 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwr5s" podUID="3308b659-80af-41d9-a040-f056900489b3" Jul 28 01:16:13.263617 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:13.263154 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:16:13.278887 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:13.278864 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:16:13.341022 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:13.340992 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zwr5s"] Jul 28 01:16:13.341202 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:13.341138 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwr5s" Jul 28 01:16:13.341282 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:13.341258 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwr5s" podUID="3308b659-80af-41d9-a040-f056900489b3" Jul 28 01:16:13.344899 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:13.344872 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8sv4q"] Jul 28 01:16:13.345064 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:13.344996 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv4q" Jul 28 01:16:13.345191 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:13.345170 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8sv4q" podUID="12e2c22a-bb31-4f97-869a-3c89eb64f837" Jul 28 01:16:13.345669 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:13.345635 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6swc2"] Jul 28 01:16:13.345762 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:13.345726 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6swc2" Jul 28 01:16:13.345826 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:13.345806 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6swc2" podUID="bcb4dce7-214c-4f64-b5af-87ca7b0d6170" Jul 28 01:16:14.265517 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:14.265440 2565 generic.go:356] "Generic (PLEG): container finished" podID="a29b525f-58ac-47f8-b759-d81742c281f4" containerID="47f73fa72e996eec71e44c296a7402a4fc09cbe60182b2d052cd907dbfea7ccc" exitCode=0 Jul 28 01:16:14.266018 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:14.265518 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2trzf" event={"ID":"a29b525f-58ac-47f8-b759-d81742c281f4","Type":"ContainerDied","Data":"47f73fa72e996eec71e44c296a7402a4fc09cbe60182b2d052cd907dbfea7ccc"} Jul 28 01:16:15.095982 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:15.095951 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6swc2" Jul 28 01:16:15.095982 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:15.095973 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwr5s" Jul 28 01:16:15.096240 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:15.095951 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv4q" Jul 28 01:16:15.096240 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:15.096087 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6swc2" podUID="bcb4dce7-214c-4f64-b5af-87ca7b0d6170" Jul 28 01:16:15.096240 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:15.096207 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8sv4q" podUID="12e2c22a-bb31-4f97-869a-3c89eb64f837" Jul 28 01:16:15.096400 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:15.096321 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwr5s" podUID="3308b659-80af-41d9-a040-f056900489b3" Jul 28 01:16:16.806074 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:16.806012 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhxk4\" (UniqueName: \"kubernetes.io/projected/bcb4dce7-214c-4f64-b5af-87ca7b0d6170-kube-api-access-hhxk4\") pod \"network-check-target-6swc2\" (UID: \"bcb4dce7-214c-4f64-b5af-87ca7b0d6170\") " pod="openshift-network-diagnostics/network-check-target-6swc2" Jul 28 01:16:16.806513 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:16.806091 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3308b659-80af-41d9-a040-f056900489b3-metrics-certs\") pod \"network-metrics-daemon-zwr5s\" (UID: \"3308b659-80af-41d9-a040-f056900489b3\") " pod="openshift-multus/network-metrics-daemon-zwr5s" Jul 28 01:16:16.806513 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:16.806140 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12e2c22a-bb31-4f97-869a-3c89eb64f837-original-pull-secret\") pod \"global-pull-secret-syncer-8sv4q\" (UID: \"12e2c22a-bb31-4f97-869a-3c89eb64f837\") " pod="kube-system/global-pull-secret-syncer-8sv4q" Jul 28 01:16:16.806513 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:16.806197 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 01:16:16.806513 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:16.806217 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 01:16:16.806513 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:16.806227 2565 projected.go:196] Error preparing data for projected volume kube-api-access-hhxk4 for pod openshift-network-diagnostics/network-check-target-6swc2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 01:16:16.806513 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:16.806240 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 01:16:16.806513 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:16.806250 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 01:16:16.806513 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:16.806280 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcb4dce7-214c-4f64-b5af-87ca7b0d6170-kube-api-access-hhxk4 podName:bcb4dce7-214c-4f64-b5af-87ca7b0d6170 nodeName:}" failed. No retries permitted until 2026-07-28 01:16:48.806262355 +0000 UTC m=+66.240660664 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hhxk4" (UniqueName: "kubernetes.io/projected/bcb4dce7-214c-4f64-b5af-87ca7b0d6170-kube-api-access-hhxk4") pod "network-check-target-6swc2" (UID: "bcb4dce7-214c-4f64-b5af-87ca7b0d6170") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 01:16:16.806513 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:16.806301 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3308b659-80af-41d9-a040-f056900489b3-metrics-certs podName:3308b659-80af-41d9-a040-f056900489b3 nodeName:}" failed. No retries permitted until 2026-07-28 01:16:48.80629302 +0000 UTC m=+66.240691312 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3308b659-80af-41d9-a040-f056900489b3-metrics-certs") pod "network-metrics-daemon-zwr5s" (UID: "3308b659-80af-41d9-a040-f056900489b3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 01:16:16.806513 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:16.806319 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12e2c22a-bb31-4f97-869a-3c89eb64f837-original-pull-secret podName:12e2c22a-bb31-4f97-869a-3c89eb64f837 nodeName:}" failed. No retries permitted until 2026-07-28 01:16:48.806309777 +0000 UTC m=+66.240708069 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/12e2c22a-bb31-4f97-869a-3c89eb64f837-original-pull-secret") pod "global-pull-secret-syncer-8sv4q" (UID: "12e2c22a-bb31-4f97-869a-3c89eb64f837") : object "kube-system"/"original-pull-secret" not registered Jul 28 01:16:17.095559 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:17.095477 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv4q" Jul 28 01:16:17.095722 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:17.095490 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwr5s" Jul 28 01:16:17.095722 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:17.095595 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8sv4q" podUID="12e2c22a-bb31-4f97-869a-3c89eb64f837" Jul 28 01:16:17.095722 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:17.095607 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6swc2" Jul 28 01:16:17.095722 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:17.095688 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zwr5s" podUID="3308b659-80af-41d9-a040-f056900489b3" Jul 28 01:16:17.095919 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:17.095758 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6swc2" podUID="bcb4dce7-214c-4f64-b5af-87ca7b0d6170" Jul 28 01:16:17.395377 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:17.395298 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-23.ec2.internal" event="NodeReady" Jul 28 01:16:17.395527 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:17.395441 2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 28 01:16:19.095423 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:19.095386 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv4q" Jul 28 01:16:19.095978 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:19.095393 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6swc2" Jul 28 01:16:19.095978 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:19.095393 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwr5s" Jul 28 01:16:19.098637 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:19.098611 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 01:16:19.098773 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:19.098650 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 01:16:19.099516 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:19.099501 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4m9v4\"" Jul 28 01:16:19.099654 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:19.099534 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 01:16:19.099654 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:19.099502 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 01:16:19.099792 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:19.099725 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j4n9n\"" Jul 28 01:16:19.280568 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:19.280537 2565 generic.go:356] "Generic (PLEG): container finished" podID="a29b525f-58ac-47f8-b759-d81742c281f4" containerID="17bd5e613e1faa8520b1a9ecc2c5d7ba582c1e2c28e0c22a38b635e3680cf163" exitCode=0 Jul 28 01:16:19.280730 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:19.280590 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2trzf" event={"ID":"a29b525f-58ac-47f8-b759-d81742c281f4","Type":"ContainerDied","Data":"17bd5e613e1faa8520b1a9ecc2c5d7ba582c1e2c28e0c22a38b635e3680cf163"} Jul 28 01:16:20.285616 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:20.285583 2565 generic.go:356] "Generic (PLEG): container finished" podID="a29b525f-58ac-47f8-b759-d81742c281f4" containerID="1164c146a8d733e3c3c0d73b12b8e0f66c3acdd615b22a8cd9f30c8483bc8b4a" exitCode=0 Jul 28 01:16:20.286108 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:20.285636 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2trzf" event={"ID":"a29b525f-58ac-47f8-b759-d81742c281f4","Type":"ContainerDied","Data":"1164c146a8d733e3c3c0d73b12b8e0f66c3acdd615b22a8cd9f30c8483bc8b4a"} Jul 28 01:16:21.290688 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:21.290454 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2trzf" event={"ID":"a29b525f-58ac-47f8-b759-d81742c281f4","Type":"ContainerStarted","Data":"ad0e4e21ac59c037fc1788a879caa1cbdef967eb7d7eab37450724632836ab49"} Jul 28 01:16:21.310660 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:21.310609 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2trzf" podStartSLOduration=9.842275861 podStartE2EDuration="38.310593456s" totalImagesPullingTime="28.468317595s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:15:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:16:21.309924232 +0000 UTC m=+38.744322547" watchObservedRunningTime="2026-07-28 01:16:21.310593456 +0000 UTC m=+38.744991770" Jul 28 01:16:35.529707 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.529670 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-m8jjv"] Jul 28 01:16:35.531786 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.531761 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-m8jjv" Jul 28 01:16:35.533309 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.533284 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6c8c86447b-8tdk7"] Jul 28 01:16:35.534771 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.534751 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 28 01:16:35.535163 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.534895 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c8c86447b-8tdk7" Jul 28 01:16:35.535163 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.535098 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 28 01:16:35.535163 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.535098 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 28 01:16:35.535163 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.535149 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-x5f5l\"" Jul 28 01:16:35.536046 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.536012 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 28 01:16:35.537651 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.537595 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-l2d29\"" Jul 28 01:16:35.537761 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.537738 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 28 01:16:35.537952 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.537932 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 28 01:16:35.538061 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.537969 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 28 01:16:35.545058 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.545019 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 28 01:16:35.546069 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.545944 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 28 01:16:35.549986 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.549962 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-m8jjv"] Jul 28 01:16:35.556021 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.555997 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c8c86447b-8tdk7"] Jul 28 01:16:35.627428 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.627397 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ftbqh"] Jul 28 01:16:35.629380 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.629363 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fdcrp"] Jul 28 01:16:35.629494 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.629478 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ftbqh" Jul 28 01:16:35.631147 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.631126 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fdcrp" Jul 28 01:16:35.631980 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.631963 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h7mrw\"" Jul 28 01:16:35.632217 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.632204 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 28 01:16:35.632279 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.632211 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/af734a7d-7584-4a47-b67a-f0b0ecfd551b-ca-trust-extracted\") pod \"image-registry-6c8c86447b-8tdk7\" (UID: \"af734a7d-7584-4a47-b67a-f0b0ecfd551b\") " pod="openshift-image-registry/image-registry-6c8c86447b-8tdk7" Jul 28 01:16:35.632279 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.632244 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fa68b140-0e24-4f1c-a9c3-7125ce3046da-trusted-ca\") pod \"console-operator-584dcb6b54-m8jjv\" (UID: \"fa68b140-0e24-4f1c-a9c3-7125ce3046da\") " pod="openshift-console-operator/console-operator-584dcb6b54-m8jjv" Jul 28 01:16:35.632279 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.632264 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/af734a7d-7584-4a47-b67a-f0b0ecfd551b-image-registry-private-configuration\") pod \"image-registry-6c8c86447b-8tdk7\" (UID: \"af734a7d-7584-4a47-b67a-f0b0ecfd551b\") " pod="openshift-image-registry/image-registry-6c8c86447b-8tdk7" Jul 28 01:16:35.632398 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.632210 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 28 01:16:35.632398 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.632333 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/af734a7d-7584-4a47-b67a-f0b0ecfd551b-bound-sa-token\") pod \"image-registry-6c8c86447b-8tdk7\" (UID: \"af734a7d-7584-4a47-b67a-f0b0ecfd551b\") " pod="openshift-image-registry/image-registry-6c8c86447b-8tdk7" Jul 28 01:16:35.632398 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.632352 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j46l\" (UniqueName: \"kubernetes.io/projected/af734a7d-7584-4a47-b67a-f0b0ecfd551b-kube-api-access-6j46l\") pod \"image-registry-6c8c86447b-8tdk7\" (UID: \"af734a7d-7584-4a47-b67a-f0b0ecfd551b\") " pod="openshift-image-registry/image-registry-6c8c86447b-8tdk7" Jul 28 01:16:35.632398 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.632379 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf9g4\" (UniqueName: \"kubernetes.io/projected/fa68b140-0e24-4f1c-a9c3-7125ce3046da-kube-api-access-pf9g4\") pod \"console-operator-584dcb6b54-m8jjv\" (UID: \"fa68b140-0e24-4f1c-a9c3-7125ce3046da\") " pod="openshift-console-operator/console-operator-584dcb6b54-m8jjv" Jul 28 01:16:35.632540 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.632428 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/af734a7d-7584-4a47-b67a-f0b0ecfd551b-installation-pull-secrets\") pod \"image-registry-6c8c86447b-8tdk7\" (UID: \"af734a7d-7584-4a47-b67a-f0b0ecfd551b\") " pod="openshift-image-registry/image-registry-6c8c86447b-8tdk7" Jul 28 01:16:35.632540 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.632493 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fa68b140-0e24-4f1c-a9c3-7125ce3046da-serving-cert\") pod \"console-operator-584dcb6b54-m8jjv\" (UID: \"fa68b140-0e24-4f1c-a9c3-7125ce3046da\") " pod="openshift-console-operator/console-operator-584dcb6b54-m8jjv" Jul 28 01:16:35.632540 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.632518 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/af734a7d-7584-4a47-b67a-f0b0ecfd551b-trusted-ca\") pod \"image-registry-6c8c86447b-8tdk7\" (UID: \"af734a7d-7584-4a47-b67a-f0b0ecfd551b\") " pod="openshift-image-registry/image-registry-6c8c86447b-8tdk7" Jul 28 01:16:35.632646 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.632578 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fa68b140-0e24-4f1c-a9c3-7125ce3046da-config\") pod \"console-operator-584dcb6b54-m8jjv\" (UID: \"fa68b140-0e24-4f1c-a9c3-7125ce3046da\") " pod="openshift-console-operator/console-operator-584dcb6b54-m8jjv" Jul 28 01:16:35.632646 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.632611 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/af734a7d-7584-4a47-b67a-f0b0ecfd551b-registry-tls\") pod \"image-registry-6c8c86447b-8tdk7\" (UID: \"af734a7d-7584-4a47-b67a-f0b0ecfd551b\") " pod="openshift-image-registry/image-registry-6c8c86447b-8tdk7" Jul 28 01:16:35.632646 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.632639 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/af734a7d-7584-4a47-b67a-f0b0ecfd551b-registry-certificates\") pod \"image-registry-6c8c86447b-8tdk7\" (UID: \"af734a7d-7584-4a47-b67a-f0b0ecfd551b\") " pod="openshift-image-registry/image-registry-6c8c86447b-8tdk7" Jul 28 01:16:35.633545 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.633530 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 28 01:16:35.633725 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.633709 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zm2x9\"" Jul 28 01:16:35.633760 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.633728 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 28 01:16:35.633971 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.633960 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 28 01:16:35.643437 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.643205 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fdcrp"] Jul 28 01:16:35.644063 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.644024 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ftbqh"] Jul 28 01:16:35.732881 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.732844 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fa68b140-0e24-4f1c-a9c3-7125ce3046da-serving-cert\") pod \"console-operator-584dcb6b54-m8jjv\" (UID: \"fa68b140-0e24-4f1c-a9c3-7125ce3046da\") " pod="openshift-console-operator/console-operator-584dcb6b54-m8jjv" Jul 28 01:16:35.732881 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.732878 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/af734a7d-7584-4a47-b67a-f0b0ecfd551b-trusted-ca\") pod \"image-registry-6c8c86447b-8tdk7\" (UID: \"af734a7d-7584-4a47-b67a-f0b0ecfd551b\") " pod="openshift-image-registry/image-registry-6c8c86447b-8tdk7" Jul 28 01:16:35.733126 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.732915 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fa68b140-0e24-4f1c-a9c3-7125ce3046da-config\") pod \"console-operator-584dcb6b54-m8jjv\" (UID: \"fa68b140-0e24-4f1c-a9c3-7125ce3046da\") " pod="openshift-console-operator/console-operator-584dcb6b54-m8jjv" Jul 28 01:16:35.733126 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.732938 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/af734a7d-7584-4a47-b67a-f0b0ecfd551b-registry-tls\") pod \"image-registry-6c8c86447b-8tdk7\" (UID: \"af734a7d-7584-4a47-b67a-f0b0ecfd551b\") " pod="openshift-image-registry/image-registry-6c8c86447b-8tdk7" Jul 28 01:16:35.733126 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.732963 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmcff\" (UniqueName: \"kubernetes.io/projected/dadc10df-28cc-4323-90c4-759cc38b55a7-kube-api-access-wmcff\") pod \"dns-default-ftbqh\" (UID: \"dadc10df-28cc-4323-90c4-759cc38b55a7\") " pod="openshift-dns/dns-default-ftbqh" Jul 28 01:16:35.733126 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.733100 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/af734a7d-7584-4a47-b67a-f0b0ecfd551b-registry-certificates\") pod \"image-registry-6c8c86447b-8tdk7\" (UID: \"af734a7d-7584-4a47-b67a-f0b0ecfd551b\") " pod="openshift-image-registry/image-registry-6c8c86447b-8tdk7" Jul 28 01:16:35.733293 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.733141 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2sm9\" (UniqueName: \"kubernetes.io/projected/88f7e7b8-110e-472b-ba6c-d2a612daaf07-kube-api-access-l2sm9\") pod \"ingress-canary-fdcrp\" (UID: \"88f7e7b8-110e-472b-ba6c-d2a612daaf07\") " pod="openshift-ingress-canary/ingress-canary-fdcrp" Jul 28 01:16:35.733293 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.733176 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/af734a7d-7584-4a47-b67a-f0b0ecfd551b-ca-trust-extracted\") pod \"image-registry-6c8c86447b-8tdk7\" (UID: \"af734a7d-7584-4a47-b67a-f0b0ecfd551b\") " pod="openshift-image-registry/image-registry-6c8c86447b-8tdk7" Jul 28 01:16:35.733293 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.733194 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88f7e7b8-110e-472b-ba6c-d2a612daaf07-cert\") pod \"ingress-canary-fdcrp\" (UID: \"88f7e7b8-110e-472b-ba6c-d2a612daaf07\") " pod="openshift-ingress-canary/ingress-canary-fdcrp" Jul 28 01:16:35.733293 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.733216 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fa68b140-0e24-4f1c-a9c3-7125ce3046da-trusted-ca\") pod \"console-operator-584dcb6b54-m8jjv\" (UID: \"fa68b140-0e24-4f1c-a9c3-7125ce3046da\") " pod="openshift-console-operator/console-operator-584dcb6b54-m8jjv" Jul 28 01:16:35.733293 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.733237 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/af734a7d-7584-4a47-b67a-f0b0ecfd551b-image-registry-private-configuration\") pod \"image-registry-6c8c86447b-8tdk7\" (UID: \"af734a7d-7584-4a47-b67a-f0b0ecfd551b\") " pod="openshift-image-registry/image-registry-6c8c86447b-8tdk7" Jul 28 01:16:35.733293 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.733256 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dadc10df-28cc-4323-90c4-759cc38b55a7-metrics-tls\") pod \"dns-default-ftbqh\" (UID: \"dadc10df-28cc-4323-90c4-759cc38b55a7\") " pod="openshift-dns/dns-default-ftbqh" Jul 28 01:16:35.733293 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.733295 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/af734a7d-7584-4a47-b67a-f0b0ecfd551b-bound-sa-token\") pod \"image-registry-6c8c86447b-8tdk7\" (UID: \"af734a7d-7584-4a47-b67a-f0b0ecfd551b\") " pod="openshift-image-registry/image-registry-6c8c86447b-8tdk7" Jul 28 01:16:35.733626 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.733325 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j46l\" (UniqueName: \"kubernetes.io/projected/af734a7d-7584-4a47-b67a-f0b0ecfd551b-kube-api-access-6j46l\") pod \"image-registry-6c8c86447b-8tdk7\" (UID: \"af734a7d-7584-4a47-b67a-f0b0ecfd551b\") " pod="openshift-image-registry/image-registry-6c8c86447b-8tdk7" Jul 28 01:16:35.733626 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.733352 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dadc10df-28cc-4323-90c4-759cc38b55a7-config-volume\") pod \"dns-default-ftbqh\" (UID: \"dadc10df-28cc-4323-90c4-759cc38b55a7\") " pod="openshift-dns/dns-default-ftbqh" Jul 28 01:16:35.733626 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.733388 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf9g4\" (UniqueName: \"kubernetes.io/projected/fa68b140-0e24-4f1c-a9c3-7125ce3046da-kube-api-access-pf9g4\") pod \"console-operator-584dcb6b54-m8jjv\" (UID: \"fa68b140-0e24-4f1c-a9c3-7125ce3046da\") " pod="openshift-console-operator/console-operator-584dcb6b54-m8jjv" Jul 28 01:16:35.733626 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.733414 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/af734a7d-7584-4a47-b67a-f0b0ecfd551b-installation-pull-secrets\") pod \"image-registry-6c8c86447b-8tdk7\" (UID: \"af734a7d-7584-4a47-b67a-f0b0ecfd551b\") " pod="openshift-image-registry/image-registry-6c8c86447b-8tdk7" Jul 28 01:16:35.733626 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.733443 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dadc10df-28cc-4323-90c4-759cc38b55a7-tmp-dir\") pod \"dns-default-ftbqh\" (UID: \"dadc10df-28cc-4323-90c4-759cc38b55a7\") " pod="openshift-dns/dns-default-ftbqh" Jul 28 01:16:35.733626 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.733611 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/af734a7d-7584-4a47-b67a-f0b0ecfd551b-ca-trust-extracted\") pod \"image-registry-6c8c86447b-8tdk7\" (UID: \"af734a7d-7584-4a47-b67a-f0b0ecfd551b\") " pod="openshift-image-registry/image-registry-6c8c86447b-8tdk7" Jul 28 01:16:35.733911 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.733810 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fa68b140-0e24-4f1c-a9c3-7125ce3046da-config\") pod \"console-operator-584dcb6b54-m8jjv\" (UID: \"fa68b140-0e24-4f1c-a9c3-7125ce3046da\") " pod="openshift-console-operator/console-operator-584dcb6b54-m8jjv" Jul 28 01:16:35.733911 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.733880 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/af734a7d-7584-4a47-b67a-f0b0ecfd551b-registry-certificates\") pod \"image-registry-6c8c86447b-8tdk7\" (UID: \"af734a7d-7584-4a47-b67a-f0b0ecfd551b\") " pod="openshift-image-registry/image-registry-6c8c86447b-8tdk7" Jul 28 01:16:35.734539 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.734517 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fa68b140-0e24-4f1c-a9c3-7125ce3046da-trusted-ca\") pod \"console-operator-584dcb6b54-m8jjv\" (UID: \"fa68b140-0e24-4f1c-a9c3-7125ce3046da\") " pod="openshift-console-operator/console-operator-584dcb6b54-m8jjv" Jul 28 01:16:35.734690 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.734545 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/af734a7d-7584-4a47-b67a-f0b0ecfd551b-trusted-ca\") pod \"image-registry-6c8c86447b-8tdk7\" (UID: \"af734a7d-7584-4a47-b67a-f0b0ecfd551b\") " pod="openshift-image-registry/image-registry-6c8c86447b-8tdk7" Jul 28 01:16:35.737472 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.737451 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fa68b140-0e24-4f1c-a9c3-7125ce3046da-serving-cert\") pod \"console-operator-584dcb6b54-m8jjv\" (UID: \"fa68b140-0e24-4f1c-a9c3-7125ce3046da\") " pod="openshift-console-operator/console-operator-584dcb6b54-m8jjv" Jul 28 01:16:35.737567 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.737548 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/af734a7d-7584-4a47-b67a-f0b0ecfd551b-image-registry-private-configuration\") pod \"image-registry-6c8c86447b-8tdk7\" (UID: \"af734a7d-7584-4a47-b67a-f0b0ecfd551b\") " pod="openshift-image-registry/image-registry-6c8c86447b-8tdk7" Jul 28 01:16:35.737567 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.737560 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/af734a7d-7584-4a47-b67a-f0b0ecfd551b-installation-pull-secrets\") pod \"image-registry-6c8c86447b-8tdk7\" (UID: \"af734a7d-7584-4a47-b67a-f0b0ecfd551b\") " pod="openshift-image-registry/image-registry-6c8c86447b-8tdk7" Jul 28 01:16:35.737697 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.737680 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/af734a7d-7584-4a47-b67a-f0b0ecfd551b-registry-tls\") pod \"image-registry-6c8c86447b-8tdk7\" (UID: \"af734a7d-7584-4a47-b67a-f0b0ecfd551b\") " pod="openshift-image-registry/image-registry-6c8c86447b-8tdk7" Jul 28 01:16:35.740906 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.740887 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/af734a7d-7584-4a47-b67a-f0b0ecfd551b-bound-sa-token\") pod \"image-registry-6c8c86447b-8tdk7\" (UID: \"af734a7d-7584-4a47-b67a-f0b0ecfd551b\") " pod="openshift-image-registry/image-registry-6c8c86447b-8tdk7" Jul 28 01:16:35.745632 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.745608 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j46l\" (UniqueName: \"kubernetes.io/projected/af734a7d-7584-4a47-b67a-f0b0ecfd551b-kube-api-access-6j46l\") pod \"image-registry-6c8c86447b-8tdk7\" (UID: \"af734a7d-7584-4a47-b67a-f0b0ecfd551b\") " pod="openshift-image-registry/image-registry-6c8c86447b-8tdk7" Jul 28 01:16:35.745949 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.745928 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf9g4\" (UniqueName: \"kubernetes.io/projected/fa68b140-0e24-4f1c-a9c3-7125ce3046da-kube-api-access-pf9g4\") pod \"console-operator-584dcb6b54-m8jjv\" (UID: \"fa68b140-0e24-4f1c-a9c3-7125ce3046da\") " pod="openshift-console-operator/console-operator-584dcb6b54-m8jjv" Jul 28 01:16:35.834449 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.834357 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dadc10df-28cc-4323-90c4-759cc38b55a7-config-volume\") pod \"dns-default-ftbqh\" (UID: \"dadc10df-28cc-4323-90c4-759cc38b55a7\") " pod="openshift-dns/dns-default-ftbqh" Jul 28 01:16:35.834449 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.834395 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dadc10df-28cc-4323-90c4-759cc38b55a7-tmp-dir\") pod \"dns-default-ftbqh\" (UID: \"dadc10df-28cc-4323-90c4-759cc38b55a7\") " pod="openshift-dns/dns-default-ftbqh" Jul 28 01:16:35.834678 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.834526 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmcff\" (UniqueName: \"kubernetes.io/projected/dadc10df-28cc-4323-90c4-759cc38b55a7-kube-api-access-wmcff\") pod \"dns-default-ftbqh\" (UID: \"dadc10df-28cc-4323-90c4-759cc38b55a7\") " pod="openshift-dns/dns-default-ftbqh" Jul 28 01:16:35.834678 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.834564 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2sm9\" (UniqueName: \"kubernetes.io/projected/88f7e7b8-110e-472b-ba6c-d2a612daaf07-kube-api-access-l2sm9\") pod \"ingress-canary-fdcrp\" (UID: \"88f7e7b8-110e-472b-ba6c-d2a612daaf07\") " pod="openshift-ingress-canary/ingress-canary-fdcrp" Jul 28 01:16:35.834678 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.834586 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88f7e7b8-110e-472b-ba6c-d2a612daaf07-cert\") pod \"ingress-canary-fdcrp\" (UID: \"88f7e7b8-110e-472b-ba6c-d2a612daaf07\") " pod="openshift-ingress-canary/ingress-canary-fdcrp" Jul 28 01:16:35.834678 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.834612 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dadc10df-28cc-4323-90c4-759cc38b55a7-metrics-tls\") pod \"dns-default-ftbqh\" (UID: \"dadc10df-28cc-4323-90c4-759cc38b55a7\") " pod="openshift-dns/dns-default-ftbqh" Jul 28 01:16:35.834893 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.834849 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dadc10df-28cc-4323-90c4-759cc38b55a7-tmp-dir\") pod \"dns-default-ftbqh\" (UID: \"dadc10df-28cc-4323-90c4-759cc38b55a7\") " pod="openshift-dns/dns-default-ftbqh" Jul 28 01:16:35.835056 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.835014 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dadc10df-28cc-4323-90c4-759cc38b55a7-config-volume\") pod \"dns-default-ftbqh\" (UID: \"dadc10df-28cc-4323-90c4-759cc38b55a7\") " pod="openshift-dns/dns-default-ftbqh" Jul 28 01:16:35.836847 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.836827 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dadc10df-28cc-4323-90c4-759cc38b55a7-metrics-tls\") pod \"dns-default-ftbqh\" (UID: \"dadc10df-28cc-4323-90c4-759cc38b55a7\") " pod="openshift-dns/dns-default-ftbqh" Jul 28 01:16:35.836986 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.836969 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88f7e7b8-110e-472b-ba6c-d2a612daaf07-cert\") pod \"ingress-canary-fdcrp\" (UID: \"88f7e7b8-110e-472b-ba6c-d2a612daaf07\") " pod="openshift-ingress-canary/ingress-canary-fdcrp" Jul 28 01:16:35.842626 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.842607 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmcff\" (UniqueName: \"kubernetes.io/projected/dadc10df-28cc-4323-90c4-759cc38b55a7-kube-api-access-wmcff\") pod \"dns-default-ftbqh\" (UID: \"dadc10df-28cc-4323-90c4-759cc38b55a7\") " pod="openshift-dns/dns-default-ftbqh" Jul 28 01:16:35.842891 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.842872 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2sm9\" (UniqueName: \"kubernetes.io/projected/88f7e7b8-110e-472b-ba6c-d2a612daaf07-kube-api-access-l2sm9\") pod \"ingress-canary-fdcrp\" (UID: \"88f7e7b8-110e-472b-ba6c-d2a612daaf07\") " pod="openshift-ingress-canary/ingress-canary-fdcrp" Jul 28 01:16:35.844580 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.844561 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-m8jjv" Jul 28 01:16:35.850688 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.850670 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c8c86447b-8tdk7" Jul 28 01:16:35.939417 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.939387 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ftbqh" Jul 28 01:16:35.944735 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.944560 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fdcrp" Jul 28 01:16:35.992475 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:35.992438 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-m8jjv"] Jul 28 01:16:35.996014 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:16:35.995881 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa68b140_0e24_4f1c_a9c3_7125ce3046da.slice/crio-027247e07017fc3a788315230c6f752365b4e516740dbaaa572dc55b1845bf9a WatchSource:0}: Error finding container 027247e07017fc3a788315230c6f752365b4e516740dbaaa572dc55b1845bf9a: Status 404 returned error can't find the container with id 027247e07017fc3a788315230c6f752365b4e516740dbaaa572dc55b1845bf9a Jul 28 01:16:36.022614 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:36.022580 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c8c86447b-8tdk7"] Jul 28 01:16:36.026168 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:16:36.026137 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf734a7d_7584_4a47_b67a_f0b0ecfd551b.slice/crio-09dffa2386fd4f2993c3a10c647c029581b63cdb4b37269d974fc9b9a4cf2390 WatchSource:0}: Error finding container 09dffa2386fd4f2993c3a10c647c029581b63cdb4b37269d974fc9b9a4cf2390: Status 404 returned error can't find the container with id 09dffa2386fd4f2993c3a10c647c029581b63cdb4b37269d974fc9b9a4cf2390 Jul 28 01:16:36.079781 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:36.079742 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ftbqh"] Jul 28 01:16:36.085347 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:16:36.085278 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddadc10df_28cc_4323_90c4_759cc38b55a7.slice/crio-00092d57fee0f5a814e8fb7cb16c6f7320c23b45b06c5907c061ab7bcae2b292 WatchSource:0}: Error finding container 00092d57fee0f5a814e8fb7cb16c6f7320c23b45b06c5907c061ab7bcae2b292: Status 404 returned error can't find the container with id 00092d57fee0f5a814e8fb7cb16c6f7320c23b45b06c5907c061ab7bcae2b292 Jul 28 01:16:36.095917 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:36.095893 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fdcrp"] Jul 28 01:16:36.099444 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:16:36.099416 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88f7e7b8_110e_472b_ba6c_d2a612daaf07.slice/crio-4ccdb1b47643b14240856a188aad333143345d83f287791afc89259aa4e27f70 WatchSource:0}: Error finding container 4ccdb1b47643b14240856a188aad333143345d83f287791afc89259aa4e27f70: Status 404 returned error can't find the container with id 4ccdb1b47643b14240856a188aad333143345d83f287791afc89259aa4e27f70 Jul 28 01:16:36.323008 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:36.322969 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fdcrp" event={"ID":"88f7e7b8-110e-472b-ba6c-d2a612daaf07","Type":"ContainerStarted","Data":"4ccdb1b47643b14240856a188aad333143345d83f287791afc89259aa4e27f70"} Jul 28 01:16:36.324495 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:36.324469 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c8c86447b-8tdk7" event={"ID":"af734a7d-7584-4a47-b67a-f0b0ecfd551b","Type":"ContainerStarted","Data":"ff9991268ed42ec7c2057dff01ef5e07c33a8cef11ea85d8f8cb69aa6aa2687e"} Jul 28 01:16:36.324667 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:36.324510 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c8c86447b-8tdk7" event={"ID":"af734a7d-7584-4a47-b67a-f0b0ecfd551b","Type":"ContainerStarted","Data":"09dffa2386fd4f2993c3a10c647c029581b63cdb4b37269d974fc9b9a4cf2390"} Jul 28 01:16:36.324667 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:36.324551 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6c8c86447b-8tdk7" Jul 28 01:16:36.325618 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:36.325592 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-m8jjv" event={"ID":"fa68b140-0e24-4f1c-a9c3-7125ce3046da","Type":"ContainerStarted","Data":"027247e07017fc3a788315230c6f752365b4e516740dbaaa572dc55b1845bf9a"} Jul 28 01:16:36.326623 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:36.326599 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ftbqh" event={"ID":"dadc10df-28cc-4323-90c4-759cc38b55a7","Type":"ContainerStarted","Data":"00092d57fee0f5a814e8fb7cb16c6f7320c23b45b06c5907c061ab7bcae2b292"} Jul 28 01:16:36.345440 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:36.345325 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6c8c86447b-8tdk7" podStartSLOduration=1.345308761 podStartE2EDuration="1.345308761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:16:36.344757292 +0000 UTC m=+53.779155609" watchObservedRunningTime="2026-07-28 01:16:36.345308761 +0000 UTC m=+53.779707075" Jul 28 01:16:39.342059 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:39.341953 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ftbqh" event={"ID":"dadc10df-28cc-4323-90c4-759cc38b55a7","Type":"ContainerStarted","Data":"9d02b7011d10a169efbd891ebc96c06015658aa3d1c02cedde89322589694510"} Jul 28 01:16:39.345889 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:39.344211 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fdcrp" event={"ID":"88f7e7b8-110e-472b-ba6c-d2a612daaf07","Type":"ContainerStarted","Data":"2a8cbc59bad7892da4f4b0b7573468ebbab3a584526938ef4997ff71fb5c3d5a"} Jul 28 01:16:39.347449 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:39.347303 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-m8jjv" event={"ID":"fa68b140-0e24-4f1c-a9c3-7125ce3046da","Type":"ContainerStarted","Data":"213d5a28b50196b0057eece94680bfc25771e47baeadf8b92b1e1c9ebc624799"} Jul 28 01:16:39.353220 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:39.350307 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-m8jjv" Jul 28 01:16:39.353220 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:39.352381 2565 patch_prober.go:28] interesting pod/console-operator-584dcb6b54-m8jjv container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.133.0.6:8443/readyz\": dial tcp 10.133.0.6:8443: connect: connection refused" start-of-body= Jul 28 01:16:39.353220 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:39.352441 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-584dcb6b54-m8jjv" podUID="fa68b140-0e24-4f1c-a9c3-7125ce3046da" containerName="console-operator" probeResult="failure" output="Get \"https://10.133.0.6:8443/readyz\": dial tcp 10.133.0.6:8443: connect: connection refused" Jul 28 01:16:39.363842 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:39.363796 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fdcrp" podStartSLOduration=1.246801154 podStartE2EDuration="4.363781024s" totalImagesPullingTime="3.11697987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:16:39.363063394 +0000 UTC m=+56.797461709" watchObservedRunningTime="2026-07-28 01:16:39.363781024 +0000 UTC m=+56.798179338" Jul 28 01:16:39.379094 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:39.379012 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-m8jjv" podStartSLOduration=1.164968736 podStartE2EDuration="4.378990595s" totalImagesPullingTime="3.214021859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:16:39.378840238 +0000 UTC m=+56.813238552" watchObservedRunningTime="2026-07-28 01:16:39.378990595 +0000 UTC m=+56.813388905" Jul 28 01:16:40.351346 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:40.351310 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ftbqh" event={"ID":"dadc10df-28cc-4323-90c4-759cc38b55a7","Type":"ContainerStarted","Data":"c7c1267c84eeba3f109d3c2101f0f2e95b4b537955b24d0b55396c2581fbba39"} Jul 28 01:16:40.355726 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:40.355700 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-m8jjv" Jul 28 01:16:40.369397 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:40.369356 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ftbqh" podStartSLOduration=2.239212824 podStartE2EDuration="5.369344202s" totalImagesPullingTime="3.130131378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:16:40.367603293 +0000 UTC m=+57.802001608" watchObservedRunningTime="2026-07-28 01:16:40.369344202 +0000 UTC m=+57.803742495" Jul 28 01:16:41.353651 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:41.353609 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ftbqh" Jul 28 01:16:45.278632 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:45.278603 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kbl47" Jul 28 01:16:47.697557 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:47.697522 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-w6xtp"] Jul 28 01:16:47.729626 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:47.729599 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w6xtp"] Jul 28 01:16:47.729791 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:47.729729 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w6xtp" Jul 28 01:16:47.741027 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:47.740998 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 28 01:16:47.741027 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:47.741014 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 28 01:16:47.741215 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:47.741020 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 28 01:16:47.741215 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:47.741075 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 28 01:16:47.741215 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:47.741193 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gz7nz\"" Jul 28 01:16:47.824562 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:47.824528 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/72e36b06-6877-4705-9477-4682cd601a21-data-volume\") pod \"insights-runtime-extractor-w6xtp\" (UID: \"72e36b06-6877-4705-9477-4682cd601a21\") " pod="openshift-insights/insights-runtime-extractor-w6xtp" Jul 28 01:16:47.824562 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:47.824566 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72e36b06-6877-4705-9477-4682cd601a21-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w6xtp\" (UID: \"72e36b06-6877-4705-9477-4682cd601a21\") " pod="openshift-insights/insights-runtime-extractor-w6xtp" Jul 28 01:16:47.824780 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:47.824585 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/72e36b06-6877-4705-9477-4682cd601a21-crio-socket\") pod \"insights-runtime-extractor-w6xtp\" (UID: \"72e36b06-6877-4705-9477-4682cd601a21\") " pod="openshift-insights/insights-runtime-extractor-w6xtp" Jul 28 01:16:47.824780 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:47.824602 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72e36b06-6877-4705-9477-4682cd601a21-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w6xtp\" (UID: \"72e36b06-6877-4705-9477-4682cd601a21\") " pod="openshift-insights/insights-runtime-extractor-w6xtp" Jul 28 01:16:47.824780 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:47.824719 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4xbp\" (UniqueName: \"kubernetes.io/projected/72e36b06-6877-4705-9477-4682cd601a21-kube-api-access-v4xbp\") pod \"insights-runtime-extractor-w6xtp\" (UID: \"72e36b06-6877-4705-9477-4682cd601a21\") " pod="openshift-insights/insights-runtime-extractor-w6xtp" Jul 28 01:16:47.925390 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:47.925352 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/72e36b06-6877-4705-9477-4682cd601a21-crio-socket\") pod \"insights-runtime-extractor-w6xtp\" (UID: \"72e36b06-6877-4705-9477-4682cd601a21\") " pod="openshift-insights/insights-runtime-extractor-w6xtp" Jul 28 01:16:47.925390 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:47.925387 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72e36b06-6877-4705-9477-4682cd601a21-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w6xtp\" (UID: \"72e36b06-6877-4705-9477-4682cd601a21\") " pod="openshift-insights/insights-runtime-extractor-w6xtp" Jul 28 01:16:47.925613 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:47.925425 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4xbp\" (UniqueName: \"kubernetes.io/projected/72e36b06-6877-4705-9477-4682cd601a21-kube-api-access-v4xbp\") pod \"insights-runtime-extractor-w6xtp\" (UID: \"72e36b06-6877-4705-9477-4682cd601a21\") " pod="openshift-insights/insights-runtime-extractor-w6xtp" Jul 28 01:16:47.925613 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:47.925487 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/72e36b06-6877-4705-9477-4682cd601a21-data-volume\") pod \"insights-runtime-extractor-w6xtp\" (UID: \"72e36b06-6877-4705-9477-4682cd601a21\") " pod="openshift-insights/insights-runtime-extractor-w6xtp" Jul 28 01:16:47.925613 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:47.925493 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/72e36b06-6877-4705-9477-4682cd601a21-crio-socket\") pod \"insights-runtime-extractor-w6xtp\" (UID: \"72e36b06-6877-4705-9477-4682cd601a21\") " pod="openshift-insights/insights-runtime-extractor-w6xtp" Jul 28 01:16:47.925613 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:47.925524 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72e36b06-6877-4705-9477-4682cd601a21-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w6xtp\" (UID: \"72e36b06-6877-4705-9477-4682cd601a21\") " pod="openshift-insights/insights-runtime-extractor-w6xtp" Jul 28 01:16:47.925613 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:47.925549 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 01:16:47.925613 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:47.925593 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 01:16:47.925613 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:47.925617 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72e36b06-6877-4705-9477-4682cd601a21-insights-runtime-extractor-tls podName:72e36b06-6877-4705-9477-4682cd601a21 nodeName:}" failed. No retries permitted until 2026-07-28 01:16:48.425597465 +0000 UTC m=+65.859995774 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/72e36b06-6877-4705-9477-4682cd601a21-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w6xtp" (UID: "72e36b06-6877-4705-9477-4682cd601a21") : secret "insights-runtime-extractor-tls" not found Jul 28 01:16:47.925904 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:47.925635 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/72e36b06-6877-4705-9477-4682cd601a21-kube-rbac-proxy-cm podName:72e36b06-6877-4705-9477-4682cd601a21 nodeName:}" failed. No retries permitted until 2026-07-28 01:16:48.425625298 +0000 UTC m=+65.860023601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/72e36b06-6877-4705-9477-4682cd601a21-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w6xtp" (UID: "72e36b06-6877-4705-9477-4682cd601a21") : configmap "kube-rbac-proxy" not found Jul 28 01:16:47.925904 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:47.925753 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/72e36b06-6877-4705-9477-4682cd601a21-data-volume\") pod \"insights-runtime-extractor-w6xtp\" (UID: \"72e36b06-6877-4705-9477-4682cd601a21\") " pod="openshift-insights/insights-runtime-extractor-w6xtp" Jul 28 01:16:47.934516 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:47.934486 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4xbp\" (UniqueName: \"kubernetes.io/projected/72e36b06-6877-4705-9477-4682cd601a21-kube-api-access-v4xbp\") pod \"insights-runtime-extractor-w6xtp\" (UID: \"72e36b06-6877-4705-9477-4682cd601a21\") " pod="openshift-insights/insights-runtime-extractor-w6xtp" Jul 28 01:16:48.430007 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:48.429967 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72e36b06-6877-4705-9477-4682cd601a21-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w6xtp\" (UID: \"72e36b06-6877-4705-9477-4682cd601a21\") " pod="openshift-insights/insights-runtime-extractor-w6xtp" Jul 28 01:16:48.430007 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:48.430010 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72e36b06-6877-4705-9477-4682cd601a21-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w6xtp\" (UID: \"72e36b06-6877-4705-9477-4682cd601a21\") " pod="openshift-insights/insights-runtime-extractor-w6xtp" Jul 28 01:16:48.430256 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:48.430097 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 01:16:48.430256 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:48.430165 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/72e36b06-6877-4705-9477-4682cd601a21-kube-rbac-proxy-cm podName:72e36b06-6877-4705-9477-4682cd601a21 nodeName:}" failed. No retries permitted until 2026-07-28 01:16:49.430149579 +0000 UTC m=+66.864547870 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/72e36b06-6877-4705-9477-4682cd601a21-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w6xtp" (UID: "72e36b06-6877-4705-9477-4682cd601a21") : configmap "kube-rbac-proxy" not found Jul 28 01:16:48.430256 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:48.430172 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 01:16:48.430256 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:48.430226 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72e36b06-6877-4705-9477-4682cd601a21-insights-runtime-extractor-tls podName:72e36b06-6877-4705-9477-4682cd601a21 nodeName:}" failed. No retries permitted until 2026-07-28 01:16:49.430212184 +0000 UTC m=+66.864610475 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/72e36b06-6877-4705-9477-4682cd601a21-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w6xtp" (UID: "72e36b06-6877-4705-9477-4682cd601a21") : secret "insights-runtime-extractor-tls" not found Jul 28 01:16:48.832772 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:48.832735 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3308b659-80af-41d9-a040-f056900489b3-metrics-certs\") pod \"network-metrics-daemon-zwr5s\" (UID: \"3308b659-80af-41d9-a040-f056900489b3\") " pod="openshift-multus/network-metrics-daemon-zwr5s" Jul 28 01:16:48.833173 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:48.832913 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12e2c22a-bb31-4f97-869a-3c89eb64f837-original-pull-secret\") pod \"global-pull-secret-syncer-8sv4q\" (UID: \"12e2c22a-bb31-4f97-869a-3c89eb64f837\") " pod="kube-system/global-pull-secret-syncer-8sv4q" Jul 28 01:16:48.833173 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:48.832974 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhxk4\" (UniqueName: \"kubernetes.io/projected/bcb4dce7-214c-4f64-b5af-87ca7b0d6170-kube-api-access-hhxk4\") pod \"network-check-target-6swc2\" (UID: \"bcb4dce7-214c-4f64-b5af-87ca7b0d6170\") " pod="openshift-network-diagnostics/network-check-target-6swc2" Jul 28 01:16:48.835315 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:48.835295 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 01:16:48.835396 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:48.835295 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 01:16:48.836419 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:48.836404 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 01:16:48.845610 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:48.845587 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12e2c22a-bb31-4f97-869a-3c89eb64f837-original-pull-secret\") pod \"global-pull-secret-syncer-8sv4q\" (UID: \"12e2c22a-bb31-4f97-869a-3c89eb64f837\") " pod="kube-system/global-pull-secret-syncer-8sv4q" Jul 28 01:16:48.845743 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:48.845728 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3308b659-80af-41d9-a040-f056900489b3-metrics-certs\") pod \"network-metrics-daemon-zwr5s\" (UID: \"3308b659-80af-41d9-a040-f056900489b3\") " pod="openshift-multus/network-metrics-daemon-zwr5s" Jul 28 01:16:48.845917 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:48.845902 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 01:16:48.855850 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:48.855828 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhxk4\" (UniqueName: \"kubernetes.io/projected/bcb4dce7-214c-4f64-b5af-87ca7b0d6170-kube-api-access-hhxk4\") pod \"network-check-target-6swc2\" (UID: \"bcb4dce7-214c-4f64-b5af-87ca7b0d6170\") " pod="openshift-network-diagnostics/network-check-target-6swc2" Jul 28 01:16:49.106090 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:49.106003 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8sv4q" Jul 28 01:16:49.115228 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:49.115194 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4m9v4\"" Jul 28 01:16:49.118808 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:49.118789 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j4n9n\"" Jul 28 01:16:49.121714 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:49.121692 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6swc2" Jul 28 01:16:49.124361 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:49.124339 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ftbqh_dadc10df-28cc-4323-90c4-759cc38b55a7/dns/0.log" Jul 28 01:16:49.126718 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:49.126694 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zwr5s" Jul 28 01:16:49.264434 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:49.264396 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8sv4q"] Jul 28 01:16:49.267241 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:16:49.267198 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12e2c22a_bb31_4f97_869a_3c89eb64f837.slice/crio-329a5f0f6c7ce7b52ab13d958b0ccefb301618323fed0fd09cc4267921443afa WatchSource:0}: Error finding container 329a5f0f6c7ce7b52ab13d958b0ccefb301618323fed0fd09cc4267921443afa: Status 404 returned error can't find the container with id 329a5f0f6c7ce7b52ab13d958b0ccefb301618323fed0fd09cc4267921443afa Jul 28 01:16:49.287659 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:49.287628 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6swc2"] Jul 28 01:16:49.290265 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:16:49.290236 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcb4dce7_214c_4f64_b5af_87ca7b0d6170.slice/crio-256936884dca1d0cb9058ab25cedf196fccf36e6a0e6fd7108cc5f02702d560a WatchSource:0}: Error finding container 256936884dca1d0cb9058ab25cedf196fccf36e6a0e6fd7108cc5f02702d560a: Status 404 returned error can't find the container with id 256936884dca1d0cb9058ab25cedf196fccf36e6a0e6fd7108cc5f02702d560a Jul 28 01:16:49.302845 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:49.302821 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ftbqh_dadc10df-28cc-4323-90c4-759cc38b55a7/kube-rbac-proxy/0.log" Jul 28 01:16:49.313232 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:49.313210 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zwr5s"] Jul 28 01:16:49.315708 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:16:49.315675 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3308b659_80af_41d9_a040_f056900489b3.slice/crio-328106397d49cefdc8076be54ef5ba5cc79016ebdca27b532dae9d4c1326d768 WatchSource:0}: Error finding container 328106397d49cefdc8076be54ef5ba5cc79016ebdca27b532dae9d4c1326d768: Status 404 returned error can't find the container with id 328106397d49cefdc8076be54ef5ba5cc79016ebdca27b532dae9d4c1326d768 Jul 28 01:16:49.372583 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:49.372489 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6swc2" event={"ID":"bcb4dce7-214c-4f64-b5af-87ca7b0d6170","Type":"ContainerStarted","Data":"256936884dca1d0cb9058ab25cedf196fccf36e6a0e6fd7108cc5f02702d560a"} Jul 28 01:16:49.373548 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:49.373522 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8sv4q" event={"ID":"12e2c22a-bb31-4f97-869a-3c89eb64f837","Type":"ContainerStarted","Data":"329a5f0f6c7ce7b52ab13d958b0ccefb301618323fed0fd09cc4267921443afa"} Jul 28 01:16:49.374527 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:49.374505 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zwr5s" event={"ID":"3308b659-80af-41d9-a040-f056900489b3","Type":"ContainerStarted","Data":"328106397d49cefdc8076be54ef5ba5cc79016ebdca27b532dae9d4c1326d768"} Jul 28 01:16:49.439200 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:49.439164 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72e36b06-6877-4705-9477-4682cd601a21-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w6xtp\" (UID: \"72e36b06-6877-4705-9477-4682cd601a21\") " pod="openshift-insights/insights-runtime-extractor-w6xtp" Jul 28 01:16:49.439348 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:49.439208 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72e36b06-6877-4705-9477-4682cd601a21-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w6xtp\" (UID: \"72e36b06-6877-4705-9477-4682cd601a21\") " pod="openshift-insights/insights-runtime-extractor-w6xtp" Jul 28 01:16:49.439348 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:49.439282 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 01:16:49.439348 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:49.439341 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 01:16:49.439348 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:49.439349 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/72e36b06-6877-4705-9477-4682cd601a21-kube-rbac-proxy-cm podName:72e36b06-6877-4705-9477-4682cd601a21 nodeName:}" failed. No retries permitted until 2026-07-28 01:16:51.439332503 +0000 UTC m=+68.873730797 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/72e36b06-6877-4705-9477-4682cd601a21-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w6xtp" (UID: "72e36b06-6877-4705-9477-4682cd601a21") : configmap "kube-rbac-proxy" not found Jul 28 01:16:49.439485 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:49.439384 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72e36b06-6877-4705-9477-4682cd601a21-insights-runtime-extractor-tls podName:72e36b06-6877-4705-9477-4682cd601a21 nodeName:}" failed. No retries permitted until 2026-07-28 01:16:51.439370138 +0000 UTC m=+68.873768431 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/72e36b06-6877-4705-9477-4682cd601a21-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w6xtp" (UID: "72e36b06-6877-4705-9477-4682cd601a21") : secret "insights-runtime-extractor-tls" not found Jul 28 01:16:49.903702 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:49.903659 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-snwqc_a97a9823-8b73-453c-9d16-8907cc6b5116/dns-node-resolver/0.log" Jul 28 01:16:50.103639 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:50.103600 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6c8c86447b-8tdk7_af734a7d-7584-4a47-b67a-f0b0ecfd551b/registry/0.log" Jul 28 01:16:50.902825 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:50.902737 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zlwvl_f32e8846-0caa-4b52-a5fd-ae65659fdea1/node-ca/0.log" Jul 28 01:16:51.358358 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:51.358323 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ftbqh" Jul 28 01:16:51.386722 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:51.386688 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zwr5s" event={"ID":"3308b659-80af-41d9-a040-f056900489b3","Type":"ContainerStarted","Data":"d5e76224ca97c0da6e584713c83ea17b05cefebecd2b14fa2b7b54169887fe07"} Jul 28 01:16:51.386878 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:51.386745 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zwr5s" event={"ID":"3308b659-80af-41d9-a040-f056900489b3","Type":"ContainerStarted","Data":"80e945d668541e3df3775b0512cb804b6f0640d9dd97ede5a27e7ff3050f8ed3"} Jul 28 01:16:51.402814 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:51.402761 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zwr5s" podStartSLOduration=67.239889194 podStartE2EDuration="1m8.402739682s" totalImagesPullingTime="1.162850488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:16:51.401249069 +0000 UTC m=+68.835647383" watchObservedRunningTime="2026-07-28 01:16:51.402739682 +0000 UTC m=+68.837137998" Jul 28 01:16:51.455671 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:51.455174 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72e36b06-6877-4705-9477-4682cd601a21-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w6xtp\" (UID: \"72e36b06-6877-4705-9477-4682cd601a21\") " pod="openshift-insights/insights-runtime-extractor-w6xtp" Jul 28 01:16:51.455671 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:51.455221 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72e36b06-6877-4705-9477-4682cd601a21-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w6xtp\" (UID: \"72e36b06-6877-4705-9477-4682cd601a21\") " pod="openshift-insights/insights-runtime-extractor-w6xtp" Jul 28 01:16:51.455671 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:51.455516 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 01:16:51.455671 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:51.455575 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72e36b06-6877-4705-9477-4682cd601a21-insights-runtime-extractor-tls podName:72e36b06-6877-4705-9477-4682cd601a21 nodeName:}" failed. No retries permitted until 2026-07-28 01:16:55.455555469 +0000 UTC m=+72.889953761 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/72e36b06-6877-4705-9477-4682cd601a21-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w6xtp" (UID: "72e36b06-6877-4705-9477-4682cd601a21") : secret "insights-runtime-extractor-tls" not found Jul 28 01:16:51.455994 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:51.455695 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 01:16:51.455994 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:16:51.455755 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/72e36b06-6877-4705-9477-4682cd601a21-kube-rbac-proxy-cm podName:72e36b06-6877-4705-9477-4682cd601a21 nodeName:}" failed. No retries permitted until 2026-07-28 01:16:55.455733908 +0000 UTC m=+72.890132212 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/72e36b06-6877-4705-9477-4682cd601a21-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w6xtp" (UID: "72e36b06-6877-4705-9477-4682cd601a21") : configmap "kube-rbac-proxy" not found Jul 28 01:16:51.703375 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:51.703339 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fdcrp_88f7e7b8-110e-472b-ba6c-d2a612daaf07/serve-healthcheck-canary/0.log" Jul 28 01:16:54.399503 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:54.399461 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8sv4q" event={"ID":"12e2c22a-bb31-4f97-869a-3c89eb64f837","Type":"ContainerStarted","Data":"09d6a78000e4f1090e440af5fc22c57f6fb07a153e4d0be776c734e2399f8e0c"} Jul 28 01:16:54.400864 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:54.400839 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6swc2" event={"ID":"bcb4dce7-214c-4f64-b5af-87ca7b0d6170","Type":"ContainerStarted","Data":"8a7bea9c8b3b92ac3e4ef6ae4d1a45cf523da4707f2e39fdd8421846f45f201d"} Jul 28 01:16:54.400983 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:54.400969 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6swc2" Jul 28 01:16:54.413431 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:54.413389 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8sv4q" podStartSLOduration=67.033137455 podStartE2EDuration="1m11.413376541s" totalImagesPullingTime="4.380239086s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:16:54.413201599 +0000 UTC m=+71.847599914" watchObservedRunningTime="2026-07-28 01:16:54.413376541 +0000 UTC m=+71.847774854" Jul 28 01:16:54.427396 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:54.427357 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6swc2" podStartSLOduration=67.073848255 podStartE2EDuration="1m11.427345247s" totalImagesPullingTime="4.353496992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:16:54.426596415 +0000 UTC m=+71.860994742" watchObservedRunningTime="2026-07-28 01:16:54.427345247 +0000 UTC m=+71.861743560" Jul 28 01:16:55.488561 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:55.488528 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72e36b06-6877-4705-9477-4682cd601a21-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w6xtp\" (UID: \"72e36b06-6877-4705-9477-4682cd601a21\") " pod="openshift-insights/insights-runtime-extractor-w6xtp" Jul 28 01:16:55.488561 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:55.488562 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72e36b06-6877-4705-9477-4682cd601a21-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w6xtp\" (UID: \"72e36b06-6877-4705-9477-4682cd601a21\") " pod="openshift-insights/insights-runtime-extractor-w6xtp" Jul 28 01:16:55.489132 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:55.489112 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72e36b06-6877-4705-9477-4682cd601a21-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w6xtp\" (UID: \"72e36b06-6877-4705-9477-4682cd601a21\") " pod="openshift-insights/insights-runtime-extractor-w6xtp" Jul 28 01:16:55.490816 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:55.490794 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72e36b06-6877-4705-9477-4682cd601a21-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w6xtp\" (UID: \"72e36b06-6877-4705-9477-4682cd601a21\") " pod="openshift-insights/insights-runtime-extractor-w6xtp" Jul 28 01:16:55.537855 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:55.537822 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w6xtp" Jul 28 01:16:55.655216 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:55.655121 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w6xtp"] Jul 28 01:16:55.657638 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:16:55.657616 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e36b06_6877_4705_9477_4682cd601a21.slice/crio-1c258d3ac6ec1c843c16ff6c19fab1b31f319e1b871a711e58fc88fc951eb942 WatchSource:0}: Error finding container 1c258d3ac6ec1c843c16ff6c19fab1b31f319e1b871a711e58fc88fc951eb942: Status 404 returned error can't find the container with id 1c258d3ac6ec1c843c16ff6c19fab1b31f319e1b871a711e58fc88fc951eb942 Jul 28 01:16:55.855521 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:55.855441 2565 patch_prober.go:28] interesting pod/image-registry-6c8c86447b-8tdk7 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 28 01:16:55.855521 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:55.855491 2565 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6c8c86447b-8tdk7" podUID="af734a7d-7584-4a47-b67a-f0b0ecfd551b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 01:16:56.325430 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:56.324962 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-cd974fb57-xvht8"] Jul 28 01:16:56.332504 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:56.332295 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cd974fb57-xvht8" Jul 28 01:16:56.337514 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:56.336668 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 28 01:16:56.337514 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:56.336911 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 28 01:16:56.337514 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:56.337109 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 28 01:16:56.337514 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:56.337211 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 28 01:16:56.337514 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:56.337362 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wnlt7\"" Jul 28 01:16:56.337514 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:56.337384 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 28 01:16:56.339086 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:56.339064 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cd974fb57-xvht8"] Jul 28 01:16:56.340476 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:56.340453 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 28 01:16:56.340980 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:56.340812 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 28 01:16:56.345201 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:56.345182 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 28 01:16:56.395571 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:56.395541 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78de07a8-46c4-4765-a675-8c8a895637d7-trusted-ca-bundle\") pod \"console-cd974fb57-xvht8\" (UID: \"78de07a8-46c4-4765-a675-8c8a895637d7\") " pod="openshift-console/console-cd974fb57-xvht8" Jul 28 01:16:56.395733 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:56.395583 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78de07a8-46c4-4765-a675-8c8a895637d7-console-config\") pod \"console-cd974fb57-xvht8\" (UID: \"78de07a8-46c4-4765-a675-8c8a895637d7\") " pod="openshift-console/console-cd974fb57-xvht8" Jul 28 01:16:56.395733 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:56.395611 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78de07a8-46c4-4765-a675-8c8a895637d7-console-oauth-config\") pod \"console-cd974fb57-xvht8\" (UID: \"78de07a8-46c4-4765-a675-8c8a895637d7\") " pod="openshift-console/console-cd974fb57-xvht8" Jul 28 01:16:56.395733 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:56.395632 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz8vg\" (UniqueName: \"kubernetes.io/projected/78de07a8-46c4-4765-a675-8c8a895637d7-kube-api-access-pz8vg\") pod \"console-cd974fb57-xvht8\" (UID: \"78de07a8-46c4-4765-a675-8c8a895637d7\") " pod="openshift-console/console-cd974fb57-xvht8" Jul 28 01:16:56.395733 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:56.395664 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78de07a8-46c4-4765-a675-8c8a895637d7-oauth-serving-cert\") pod \"console-cd974fb57-xvht8\" (UID: \"78de07a8-46c4-4765-a675-8c8a895637d7\") " pod="openshift-console/console-cd974fb57-xvht8" Jul 28 01:16:56.395733 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:56.395708 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78de07a8-46c4-4765-a675-8c8a895637d7-service-ca\") pod \"console-cd974fb57-xvht8\" (UID: \"78de07a8-46c4-4765-a675-8c8a895637d7\") " pod="openshift-console/console-cd974fb57-xvht8" Jul 28 01:16:56.395931 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:56.395732 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78de07a8-46c4-4765-a675-8c8a895637d7-console-serving-cert\") pod \"console-cd974fb57-xvht8\" (UID: \"78de07a8-46c4-4765-a675-8c8a895637d7\") " pod="openshift-console/console-cd974fb57-xvht8" Jul 28 01:16:56.409789 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:56.409756 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w6xtp" event={"ID":"72e36b06-6877-4705-9477-4682cd601a21","Type":"ContainerStarted","Data":"1c258d3ac6ec1c843c16ff6c19fab1b31f319e1b871a711e58fc88fc951eb942"} Jul 28 01:16:56.496502 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:56.496464 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78de07a8-46c4-4765-a675-8c8a895637d7-service-ca\") pod \"console-cd974fb57-xvht8\" (UID: \"78de07a8-46c4-4765-a675-8c8a895637d7\") " pod="openshift-console/console-cd974fb57-xvht8" Jul 28 01:16:56.496973 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:56.496516 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78de07a8-46c4-4765-a675-8c8a895637d7-console-serving-cert\") pod \"console-cd974fb57-xvht8\" (UID: \"78de07a8-46c4-4765-a675-8c8a895637d7\") " pod="openshift-console/console-cd974fb57-xvht8" Jul 28 01:16:56.496973 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:56.496571 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78de07a8-46c4-4765-a675-8c8a895637d7-trusted-ca-bundle\") pod \"console-cd974fb57-xvht8\" (UID: \"78de07a8-46c4-4765-a675-8c8a895637d7\") " pod="openshift-console/console-cd974fb57-xvht8" Jul 28 01:16:56.496973 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:56.496600 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78de07a8-46c4-4765-a675-8c8a895637d7-console-config\") pod \"console-cd974fb57-xvht8\" (UID: \"78de07a8-46c4-4765-a675-8c8a895637d7\") " pod="openshift-console/console-cd974fb57-xvht8" Jul 28 01:16:56.496973 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:56.496626 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78de07a8-46c4-4765-a675-8c8a895637d7-console-oauth-config\") pod \"console-cd974fb57-xvht8\" (UID: \"78de07a8-46c4-4765-a675-8c8a895637d7\") " pod="openshift-console/console-cd974fb57-xvht8" Jul 28 01:16:56.496973 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:56.496651 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz8vg\" (UniqueName: \"kubernetes.io/projected/78de07a8-46c4-4765-a675-8c8a895637d7-kube-api-access-pz8vg\") pod \"console-cd974fb57-xvht8\" (UID: \"78de07a8-46c4-4765-a675-8c8a895637d7\") " pod="openshift-console/console-cd974fb57-xvht8" Jul 28 01:16:56.496973 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:56.496689 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78de07a8-46c4-4765-a675-8c8a895637d7-oauth-serving-cert\") pod \"console-cd974fb57-xvht8\" (UID: \"78de07a8-46c4-4765-a675-8c8a895637d7\") " pod="openshift-console/console-cd974fb57-xvht8" Jul 28 01:16:56.497829 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:56.497391 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78de07a8-46c4-4765-a675-8c8a895637d7-service-ca\") pod \"console-cd974fb57-xvht8\" (UID: \"78de07a8-46c4-4765-a675-8c8a895637d7\") " pod="openshift-console/console-cd974fb57-xvht8" Jul 28 01:16:56.497829 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:56.497398 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78de07a8-46c4-4765-a675-8c8a895637d7-oauth-serving-cert\") pod \"console-cd974fb57-xvht8\" (UID: \"78de07a8-46c4-4765-a675-8c8a895637d7\") " pod="openshift-console/console-cd974fb57-xvht8" Jul 28 01:16:56.497829 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:56.497485 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78de07a8-46c4-4765-a675-8c8a895637d7-console-config\") pod \"console-cd974fb57-xvht8\" (UID: \"78de07a8-46c4-4765-a675-8c8a895637d7\") " pod="openshift-console/console-cd974fb57-xvht8" Jul 28 01:16:56.498562 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:56.498527 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78de07a8-46c4-4765-a675-8c8a895637d7-trusted-ca-bundle\") pod \"console-cd974fb57-xvht8\" (UID: \"78de07a8-46c4-4765-a675-8c8a895637d7\") " pod="openshift-console/console-cd974fb57-xvht8" Jul 28 01:16:56.500102 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:56.500078 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78de07a8-46c4-4765-a675-8c8a895637d7-console-serving-cert\") pod \"console-cd974fb57-xvht8\" (UID: \"78de07a8-46c4-4765-a675-8c8a895637d7\") " pod="openshift-console/console-cd974fb57-xvht8" Jul 28 01:16:56.500527 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:56.500506 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78de07a8-46c4-4765-a675-8c8a895637d7-console-oauth-config\") pod \"console-cd974fb57-xvht8\" (UID: \"78de07a8-46c4-4765-a675-8c8a895637d7\") " pod="openshift-console/console-cd974fb57-xvht8" Jul 28 01:16:56.506503 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:56.506478 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz8vg\" (UniqueName: \"kubernetes.io/projected/78de07a8-46c4-4765-a675-8c8a895637d7-kube-api-access-pz8vg\") pod \"console-cd974fb57-xvht8\" (UID: \"78de07a8-46c4-4765-a675-8c8a895637d7\") " pod="openshift-console/console-cd974fb57-xvht8" Jul 28 01:16:56.649267 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:56.649179 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cd974fb57-xvht8" Jul 28 01:16:56.794292 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:56.794251 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cd974fb57-xvht8"] Jul 28 01:16:56.799447 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:16:56.799414 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78de07a8_46c4_4765_a675_8c8a895637d7.slice/crio-2c3c9a4a9cb5b8b471a4cdb103c78d3b51cfc6eefb50f623d1135302a2e9a727 WatchSource:0}: Error finding container 2c3c9a4a9cb5b8b471a4cdb103c78d3b51cfc6eefb50f623d1135302a2e9a727: Status 404 returned error can't find the container with id 2c3c9a4a9cb5b8b471a4cdb103c78d3b51cfc6eefb50f623d1135302a2e9a727 Jul 28 01:16:57.336977 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:57.336200 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6c8c86447b-8tdk7" Jul 28 01:16:57.416644 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:16:57.416611 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cd974fb57-xvht8" event={"ID":"78de07a8-46c4-4765-a675-8c8a895637d7","Type":"ContainerStarted","Data":"2c3c9a4a9cb5b8b471a4cdb103c78d3b51cfc6eefb50f623d1135302a2e9a727"} Jul 28 01:17:02.443119 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:02.443079 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w6xtp" event={"ID":"72e36b06-6877-4705-9477-4682cd601a21","Type":"ContainerStarted","Data":"1ca62149c88e7dddfda36cc6ffa5001368f2d514f0348883bb1ce358f639fa85"} Jul 28 01:17:02.444887 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:02.444852 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cd974fb57-xvht8" event={"ID":"78de07a8-46c4-4765-a675-8c8a895637d7","Type":"ContainerStarted","Data":"77d7e770878a303250654a7ec9148715801f29ab602ed115894d7cd1e16012a3"} Jul 28 01:17:02.463763 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:02.463653 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-cd974fb57-xvht8" podStartSLOduration=1.4247626 podStartE2EDuration="6.463632921s" totalImagesPullingTime="5.038870321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:16:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:17:02.462475826 +0000 UTC m=+79.896874140" watchObservedRunningTime="2026-07-28 01:17:02.463632921 +0000 UTC m=+79.898031236" Jul 28 01:17:05.458953 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:05.458919 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w6xtp" event={"ID":"72e36b06-6877-4705-9477-4682cd601a21","Type":"ContainerStarted","Data":"2808b2c24ec5f3a4927ee5515254da7a73b61644808b096e36b5c7ffe9f45b92"} Jul 28 01:17:05.559992 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:05.559913 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qkgnd"] Jul 28 01:17:05.563141 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:05.563126 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qkgnd" Jul 28 01:17:05.565540 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:05.565516 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 28 01:17:05.567042 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:05.567011 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 28 01:17:05.567042 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:05.567022 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 28 01:17:05.567216 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:05.567047 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7fx46\"" Jul 28 01:17:05.567324 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:05.567309 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 28 01:17:05.567424 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:05.567408 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 28 01:17:05.567507 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:05.567492 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 28 01:17:05.679081 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:05.679027 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/94413823-494b-477d-9cc9-51f4bcb2e16f-node-exporter-accelerators-collector-config\") pod \"node-exporter-qkgnd\" (UID: \"94413823-494b-477d-9cc9-51f4bcb2e16f\") " pod="openshift-monitoring/node-exporter-qkgnd" Jul 28 01:17:05.679225 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:05.679091 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94413823-494b-477d-9cc9-51f4bcb2e16f-sys\") pod \"node-exporter-qkgnd\" (UID: \"94413823-494b-477d-9cc9-51f4bcb2e16f\") " pod="openshift-monitoring/node-exporter-qkgnd" Jul 28 01:17:05.679225 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:05.679160 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/94413823-494b-477d-9cc9-51f4bcb2e16f-node-exporter-tls\") pod \"node-exporter-qkgnd\" (UID: \"94413823-494b-477d-9cc9-51f4bcb2e16f\") " pod="openshift-monitoring/node-exporter-qkgnd" Jul 28 01:17:05.679225 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:05.679187 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw7k8\" (UniqueName: \"kubernetes.io/projected/94413823-494b-477d-9cc9-51f4bcb2e16f-kube-api-access-cw7k8\") pod \"node-exporter-qkgnd\" (UID: \"94413823-494b-477d-9cc9-51f4bcb2e16f\") " pod="openshift-monitoring/node-exporter-qkgnd" Jul 28 01:17:05.679225 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:05.679215 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/94413823-494b-477d-9cc9-51f4bcb2e16f-node-exporter-wtmp\") pod \"node-exporter-qkgnd\" (UID: \"94413823-494b-477d-9cc9-51f4bcb2e16f\") " pod="openshift-monitoring/node-exporter-qkgnd" Jul 28 01:17:05.679356 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:05.679247 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/94413823-494b-477d-9cc9-51f4bcb2e16f-node-exporter-textfile\") pod \"node-exporter-qkgnd\" (UID: \"94413823-494b-477d-9cc9-51f4bcb2e16f\") " pod="openshift-monitoring/node-exporter-qkgnd" Jul 28 01:17:05.679356 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:05.679260 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94413823-494b-477d-9cc9-51f4bcb2e16f-metrics-client-ca\") pod \"node-exporter-qkgnd\" (UID: \"94413823-494b-477d-9cc9-51f4bcb2e16f\") " pod="openshift-monitoring/node-exporter-qkgnd" Jul 28 01:17:05.679356 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:05.679280 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/94413823-494b-477d-9cc9-51f4bcb2e16f-root\") pod \"node-exporter-qkgnd\" (UID: \"94413823-494b-477d-9cc9-51f4bcb2e16f\") " pod="openshift-monitoring/node-exporter-qkgnd" Jul 28 01:17:05.679356 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:05.679316 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/94413823-494b-477d-9cc9-51f4bcb2e16f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qkgnd\" (UID: \"94413823-494b-477d-9cc9-51f4bcb2e16f\") " pod="openshift-monitoring/node-exporter-qkgnd" Jul 28 01:17:05.780663 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:05.780613 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/94413823-494b-477d-9cc9-51f4bcb2e16f-node-exporter-wtmp\") pod \"node-exporter-qkgnd\" (UID: \"94413823-494b-477d-9cc9-51f4bcb2e16f\") " pod="openshift-monitoring/node-exporter-qkgnd" Jul 28 01:17:05.780663 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:05.780666 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/94413823-494b-477d-9cc9-51f4bcb2e16f-node-exporter-textfile\") pod \"node-exporter-qkgnd\" (UID: \"94413823-494b-477d-9cc9-51f4bcb2e16f\") " pod="openshift-monitoring/node-exporter-qkgnd" Jul 28 01:17:05.780889 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:05.780788 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94413823-494b-477d-9cc9-51f4bcb2e16f-metrics-client-ca\") pod \"node-exporter-qkgnd\" (UID: \"94413823-494b-477d-9cc9-51f4bcb2e16f\") " pod="openshift-monitoring/node-exporter-qkgnd" Jul 28 01:17:05.780889 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:05.780834 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/94413823-494b-477d-9cc9-51f4bcb2e16f-root\") pod \"node-exporter-qkgnd\" (UID: \"94413823-494b-477d-9cc9-51f4bcb2e16f\") " pod="openshift-monitoring/node-exporter-qkgnd" Jul 28 01:17:05.780889 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:05.780792 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/94413823-494b-477d-9cc9-51f4bcb2e16f-node-exporter-wtmp\") pod \"node-exporter-qkgnd\" (UID: \"94413823-494b-477d-9cc9-51f4bcb2e16f\") " pod="openshift-monitoring/node-exporter-qkgnd" Jul 28 01:17:05.781073 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:05.780890 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/94413823-494b-477d-9cc9-51f4bcb2e16f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qkgnd\" (UID: \"94413823-494b-477d-9cc9-51f4bcb2e16f\") " pod="openshift-monitoring/node-exporter-qkgnd" Jul 28 01:17:05.781073 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:05.780893 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/94413823-494b-477d-9cc9-51f4bcb2e16f-root\") pod \"node-exporter-qkgnd\" (UID: \"94413823-494b-477d-9cc9-51f4bcb2e16f\") " pod="openshift-monitoring/node-exporter-qkgnd" Jul 28 01:17:05.781073 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:05.780934 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/94413823-494b-477d-9cc9-51f4bcb2e16f-node-exporter-accelerators-collector-config\") pod \"node-exporter-qkgnd\" (UID: \"94413823-494b-477d-9cc9-51f4bcb2e16f\") " pod="openshift-monitoring/node-exporter-qkgnd" Jul 28 01:17:05.781073 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:05.780956 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94413823-494b-477d-9cc9-51f4bcb2e16f-sys\") pod \"node-exporter-qkgnd\" (UID: \"94413823-494b-477d-9cc9-51f4bcb2e16f\") " pod="openshift-monitoring/node-exporter-qkgnd" Jul 28 01:17:05.781073 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:05.780981 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/94413823-494b-477d-9cc9-51f4bcb2e16f-node-exporter-tls\") pod \"node-exporter-qkgnd\" (UID: \"94413823-494b-477d-9cc9-51f4bcb2e16f\") " pod="openshift-monitoring/node-exporter-qkgnd" Jul 28 01:17:05.781073 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:05.781003 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw7k8\" (UniqueName: \"kubernetes.io/projected/94413823-494b-477d-9cc9-51f4bcb2e16f-kube-api-access-cw7k8\") pod \"node-exporter-qkgnd\" (UID: \"94413823-494b-477d-9cc9-51f4bcb2e16f\") " pod="openshift-monitoring/node-exporter-qkgnd" Jul 28 01:17:05.781073 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:05.781017 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/94413823-494b-477d-9cc9-51f4bcb2e16f-node-exporter-textfile\") pod \"node-exporter-qkgnd\" (UID: \"94413823-494b-477d-9cc9-51f4bcb2e16f\") " pod="openshift-monitoring/node-exporter-qkgnd" Jul 28 01:17:05.781387 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:05.781080 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94413823-494b-477d-9cc9-51f4bcb2e16f-sys\") pod \"node-exporter-qkgnd\" (UID: \"94413823-494b-477d-9cc9-51f4bcb2e16f\") " pod="openshift-monitoring/node-exporter-qkgnd" Jul 28 01:17:05.781387 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:17:05.781149 2565 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 28 01:17:05.781387 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:17:05.781223 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94413823-494b-477d-9cc9-51f4bcb2e16f-node-exporter-tls podName:94413823-494b-477d-9cc9-51f4bcb2e16f nodeName:}" failed. No retries permitted until 2026-07-28 01:17:06.281201464 +0000 UTC m=+83.715599762 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/94413823-494b-477d-9cc9-51f4bcb2e16f-node-exporter-tls") pod "node-exporter-qkgnd" (UID: "94413823-494b-477d-9cc9-51f4bcb2e16f") : secret "node-exporter-tls" not found Jul 28 01:17:05.781387 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:05.781338 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94413823-494b-477d-9cc9-51f4bcb2e16f-metrics-client-ca\") pod \"node-exporter-qkgnd\" (UID: \"94413823-494b-477d-9cc9-51f4bcb2e16f\") " pod="openshift-monitoring/node-exporter-qkgnd" Jul 28 01:17:05.781522 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:05.781488 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/94413823-494b-477d-9cc9-51f4bcb2e16f-node-exporter-accelerators-collector-config\") pod \"node-exporter-qkgnd\" (UID: \"94413823-494b-477d-9cc9-51f4bcb2e16f\") " pod="openshift-monitoring/node-exporter-qkgnd" Jul 28 01:17:05.783089 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:05.783071 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/94413823-494b-477d-9cc9-51f4bcb2e16f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qkgnd\" (UID: \"94413823-494b-477d-9cc9-51f4bcb2e16f\") " pod="openshift-monitoring/node-exporter-qkgnd" Jul 28 01:17:05.789072 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:05.789054 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw7k8\" (UniqueName: \"kubernetes.io/projected/94413823-494b-477d-9cc9-51f4bcb2e16f-kube-api-access-cw7k8\") pod \"node-exporter-qkgnd\" (UID: \"94413823-494b-477d-9cc9-51f4bcb2e16f\") " pod="openshift-monitoring/node-exporter-qkgnd" Jul 28 01:17:06.284572 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.284529 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/94413823-494b-477d-9cc9-51f4bcb2e16f-node-exporter-tls\") pod \"node-exporter-qkgnd\" (UID: \"94413823-494b-477d-9cc9-51f4bcb2e16f\") " pod="openshift-monitoring/node-exporter-qkgnd" Jul 28 01:17:06.284762 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:17:06.284693 2565 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 28 01:17:06.284825 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:17:06.284779 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94413823-494b-477d-9cc9-51f4bcb2e16f-node-exporter-tls podName:94413823-494b-477d-9cc9-51f4bcb2e16f nodeName:}" failed. No retries permitted until 2026-07-28 01:17:07.284758419 +0000 UTC m=+84.719156724 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/94413823-494b-477d-9cc9-51f4bcb2e16f-node-exporter-tls") pod "node-exporter-qkgnd" (UID: "94413823-494b-477d-9cc9-51f4bcb2e16f") : secret "node-exporter-tls" not found Jul 28 01:17:06.587504 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.587408 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 01:17:06.591345 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.591322 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.596410 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.596389 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 28 01:17:06.596522 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.596391 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 28 01:17:06.596584 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.596392 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 28 01:17:06.596814 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.596797 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-444q2\"" Jul 28 01:17:06.596929 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.596891 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 28 01:17:06.597020 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.597005 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 28 01:17:06.597252 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.597128 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 28 01:17:06.597599 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.597529 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 28 01:17:06.597699 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.597625 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 28 01:17:06.597807 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.597785 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 28 01:17:06.606158 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.606139 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 01:17:06.649706 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.649667 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cd974fb57-xvht8" Jul 28 01:17:06.649887 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.649733 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cd974fb57-xvht8" Jul 28 01:17:06.655162 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.655137 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-cd974fb57-xvht8" Jul 28 01:17:06.687799 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.687764 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.687799 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.687811 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.688136 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.687883 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.688136 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.687913 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e290194-0951-4cc1-b02e-9f40fcf53cf0-config-out\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.688136 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.687943 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-web-config\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.688136 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.687995 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e290194-0951-4cc1-b02e-9f40fcf53cf0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.688136 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.688013 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e290194-0951-4cc1-b02e-9f40fcf53cf0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.688136 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.688066 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-config-volume\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.688136 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.688085 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88qmm\" (UniqueName: \"kubernetes.io/projected/8e290194-0951-4cc1-b02e-9f40fcf53cf0-kube-api-access-88qmm\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.688136 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.688114 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.688136 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.688140 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8e290194-0951-4cc1-b02e-9f40fcf53cf0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.688597 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.688271 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e290194-0951-4cc1-b02e-9f40fcf53cf0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.688597 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.688325 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.789556 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.789515 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.789734 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.789567 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8e290194-0951-4cc1-b02e-9f40fcf53cf0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.789734 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.789644 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e290194-0951-4cc1-b02e-9f40fcf53cf0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.790151 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.790085 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8e290194-0951-4cc1-b02e-9f40fcf53cf0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.790151 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.790124 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.790349 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.790175 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.790349 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.790205 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.790349 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:17:06.790275 2565 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 28 01:17:06.790349 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:17:06.790339 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-secret-alertmanager-main-tls podName:8e290194-0951-4cc1-b02e-9f40fcf53cf0 nodeName:}" failed. No retries permitted until 2026-07-28 01:17:07.29031796 +0000 UTC m=+84.724716258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "8e290194-0951-4cc1-b02e-9f40fcf53cf0") : secret "alertmanager-main-tls" not found Jul 28 01:17:06.790566 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.790275 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.790621 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.790598 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e290194-0951-4cc1-b02e-9f40fcf53cf0-config-out\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.790673 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.790645 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-web-config\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.790673 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.790661 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e290194-0951-4cc1-b02e-9f40fcf53cf0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.790768 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.790675 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e290194-0951-4cc1-b02e-9f40fcf53cf0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.790768 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.790719 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e290194-0951-4cc1-b02e-9f40fcf53cf0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.790768 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.790755 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-config-volume\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.790908 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.790782 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88qmm\" (UniqueName: \"kubernetes.io/projected/8e290194-0951-4cc1-b02e-9f40fcf53cf0-kube-api-access-88qmm\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.791965 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.791430 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e290194-0951-4cc1-b02e-9f40fcf53cf0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.793677 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.793427 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.793677 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.793549 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.793825 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.793678 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e290194-0951-4cc1-b02e-9f40fcf53cf0-config-out\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.794178 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.794150 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.794332 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.794306 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e290194-0951-4cc1-b02e-9f40fcf53cf0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.794332 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.794321 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-web-config\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.794932 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.794848 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-config-volume\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.795322 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.795281 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:06.799060 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:06.799027 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88qmm\" (UniqueName: \"kubernetes.io/projected/8e290194-0951-4cc1-b02e-9f40fcf53cf0-kube-api-access-88qmm\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:07.295132 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:07.295091 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:07.295327 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:07.295157 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/94413823-494b-477d-9cc9-51f4bcb2e16f-node-exporter-tls\") pod \"node-exporter-qkgnd\" (UID: \"94413823-494b-477d-9cc9-51f4bcb2e16f\") " pod="openshift-monitoring/node-exporter-qkgnd" Jul 28 01:17:07.297535 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:07.297512 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/94413823-494b-477d-9cc9-51f4bcb2e16f-node-exporter-tls\") pod \"node-exporter-qkgnd\" (UID: \"94413823-494b-477d-9cc9-51f4bcb2e16f\") " pod="openshift-monitoring/node-exporter-qkgnd" Jul 28 01:17:07.297814 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:07.297792 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:07.372021 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:07.371979 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qkgnd" Jul 28 01:17:07.471142 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:07.471112 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cd974fb57-xvht8" Jul 28 01:17:07.503131 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:07.503103 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:17:08.827232 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:17:08.827205 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94413823_494b_477d_9cc9_51f4bcb2e16f.slice/crio-171c3af85281184440ef7617de4203a2e1ed0082b92ac313d2e3ae1439f6825f WatchSource:0}: Error finding container 171c3af85281184440ef7617de4203a2e1ed0082b92ac313d2e3ae1439f6825f: Status 404 returned error can't find the container with id 171c3af85281184440ef7617de4203a2e1ed0082b92ac313d2e3ae1439f6825f Jul 28 01:17:08.955302 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:08.955278 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 01:17:08.956940 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:17:08.956915 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e290194_0951_4cc1_b02e_9f40fcf53cf0.slice/crio-5bc8eab696d5799746c79a9a453d8af6deaf0d63416726c71faf3d840c1e4600 WatchSource:0}: Error finding container 5bc8eab696d5799746c79a9a453d8af6deaf0d63416726c71faf3d840c1e4600: Status 404 returned error can't find the container with id 5bc8eab696d5799746c79a9a453d8af6deaf0d63416726c71faf3d840c1e4600 Jul 28 01:17:09.474947 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:09.474911 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e290194-0951-4cc1-b02e-9f40fcf53cf0","Type":"ContainerStarted","Data":"5bc8eab696d5799746c79a9a453d8af6deaf0d63416726c71faf3d840c1e4600"} Jul 28 01:17:09.476082 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:09.476055 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qkgnd" event={"ID":"94413823-494b-477d-9cc9-51f4bcb2e16f","Type":"ContainerStarted","Data":"171c3af85281184440ef7617de4203a2e1ed0082b92ac313d2e3ae1439f6825f"} Jul 28 01:17:09.477820 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:09.477801 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w6xtp" event={"ID":"72e36b06-6877-4705-9477-4682cd601a21","Type":"ContainerStarted","Data":"56a69b8cc4ae14b91d1d544db6423c3e3be9dd6f4315b84dc1b5db65fbfe7f95"} Jul 28 01:17:09.495438 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:09.495392 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-w6xtp" podStartSLOduration=9.420981979 podStartE2EDuration="22.495377999s" totalImagesPullingTime="13.07439602s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:16:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:17:09.4935667 +0000 UTC m=+86.927965014" watchObservedRunningTime="2026-07-28 01:17:09.495377999 +0000 UTC m=+86.929776312" Jul 28 01:17:09.825472 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:09.825440 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6465894bc4-lxqgl"] Jul 28 01:17:09.828626 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:09.828603 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6465894bc4-lxqgl" Jul 28 01:17:09.831153 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:09.831128 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 28 01:17:09.832960 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:09.832669 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-dgbf6\"" Jul 28 01:17:09.832960 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:09.832669 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 28 01:17:09.832960 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:09.832672 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-cvd613vg74b2o\"" Jul 28 01:17:09.832960 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:09.832736 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 28 01:17:09.832960 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:09.832672 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 28 01:17:09.840544 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:09.840524 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6465894bc4-lxqgl"] Jul 28 01:17:09.916569 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:09.916536 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/51fdc520-b7b9-45fc-b93f-c220cab87fa9-secret-metrics-server-tls\") pod \"metrics-server-6465894bc4-lxqgl\" (UID: \"51fdc520-b7b9-45fc-b93f-c220cab87fa9\") " pod="openshift-monitoring/metrics-server-6465894bc4-lxqgl" Jul 28 01:17:09.916569 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:09.916576 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/51fdc520-b7b9-45fc-b93f-c220cab87fa9-metrics-server-audit-profiles\") pod \"metrics-server-6465894bc4-lxqgl\" (UID: \"51fdc520-b7b9-45fc-b93f-c220cab87fa9\") " pod="openshift-monitoring/metrics-server-6465894bc4-lxqgl" Jul 28 01:17:09.916788 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:09.916627 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/51fdc520-b7b9-45fc-b93f-c220cab87fa9-client-ca-bundle\") pod \"metrics-server-6465894bc4-lxqgl\" (UID: \"51fdc520-b7b9-45fc-b93f-c220cab87fa9\") " pod="openshift-monitoring/metrics-server-6465894bc4-lxqgl" Jul 28 01:17:09.916788 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:09.916660 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/51fdc520-b7b9-45fc-b93f-c220cab87fa9-audit-log\") pod \"metrics-server-6465894bc4-lxqgl\" (UID: \"51fdc520-b7b9-45fc-b93f-c220cab87fa9\") " pod="openshift-monitoring/metrics-server-6465894bc4-lxqgl" Jul 28 01:17:09.916788 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:09.916746 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51fdc520-b7b9-45fc-b93f-c220cab87fa9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6465894bc4-lxqgl\" (UID: \"51fdc520-b7b9-45fc-b93f-c220cab87fa9\") " pod="openshift-monitoring/metrics-server-6465894bc4-lxqgl" Jul 28 01:17:09.916949 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:09.916794 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/51fdc520-b7b9-45fc-b93f-c220cab87fa9-secret-metrics-server-client-certs\") pod \"metrics-server-6465894bc4-lxqgl\" (UID: \"51fdc520-b7b9-45fc-b93f-c220cab87fa9\") " pod="openshift-monitoring/metrics-server-6465894bc4-lxqgl" Jul 28 01:17:09.916949 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:09.916865 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2gdw\" (UniqueName: \"kubernetes.io/projected/51fdc520-b7b9-45fc-b93f-c220cab87fa9-kube-api-access-b2gdw\") pod \"metrics-server-6465894bc4-lxqgl\" (UID: \"51fdc520-b7b9-45fc-b93f-c220cab87fa9\") " pod="openshift-monitoring/metrics-server-6465894bc4-lxqgl" Jul 28 01:17:10.017707 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.017623 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/51fdc520-b7b9-45fc-b93f-c220cab87fa9-secret-metrics-server-tls\") pod \"metrics-server-6465894bc4-lxqgl\" (UID: \"51fdc520-b7b9-45fc-b93f-c220cab87fa9\") " pod="openshift-monitoring/metrics-server-6465894bc4-lxqgl" Jul 28 01:17:10.017707 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.017666 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/51fdc520-b7b9-45fc-b93f-c220cab87fa9-metrics-server-audit-profiles\") pod \"metrics-server-6465894bc4-lxqgl\" (UID: \"51fdc520-b7b9-45fc-b93f-c220cab87fa9\") " pod="openshift-monitoring/metrics-server-6465894bc4-lxqgl" Jul 28 01:17:10.017926 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.017716 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/51fdc520-b7b9-45fc-b93f-c220cab87fa9-client-ca-bundle\") pod \"metrics-server-6465894bc4-lxqgl\" (UID: \"51fdc520-b7b9-45fc-b93f-c220cab87fa9\") " pod="openshift-monitoring/metrics-server-6465894bc4-lxqgl" Jul 28 01:17:10.017926 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.017742 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/51fdc520-b7b9-45fc-b93f-c220cab87fa9-audit-log\") pod \"metrics-server-6465894bc4-lxqgl\" (UID: \"51fdc520-b7b9-45fc-b93f-c220cab87fa9\") " pod="openshift-monitoring/metrics-server-6465894bc4-lxqgl" Jul 28 01:17:10.017926 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.017822 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51fdc520-b7b9-45fc-b93f-c220cab87fa9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6465894bc4-lxqgl\" (UID: \"51fdc520-b7b9-45fc-b93f-c220cab87fa9\") " pod="openshift-monitoring/metrics-server-6465894bc4-lxqgl" Jul 28 01:17:10.017926 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.017857 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/51fdc520-b7b9-45fc-b93f-c220cab87fa9-secret-metrics-server-client-certs\") pod \"metrics-server-6465894bc4-lxqgl\" (UID: \"51fdc520-b7b9-45fc-b93f-c220cab87fa9\") " pod="openshift-monitoring/metrics-server-6465894bc4-lxqgl" Jul 28 01:17:10.017926 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.017889 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2gdw\" (UniqueName: \"kubernetes.io/projected/51fdc520-b7b9-45fc-b93f-c220cab87fa9-kube-api-access-b2gdw\") pod \"metrics-server-6465894bc4-lxqgl\" (UID: \"51fdc520-b7b9-45fc-b93f-c220cab87fa9\") " pod="openshift-monitoring/metrics-server-6465894bc4-lxqgl" Jul 28 01:17:10.018540 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.018471 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/51fdc520-b7b9-45fc-b93f-c220cab87fa9-audit-log\") pod \"metrics-server-6465894bc4-lxqgl\" (UID: \"51fdc520-b7b9-45fc-b93f-c220cab87fa9\") " pod="openshift-monitoring/metrics-server-6465894bc4-lxqgl" Jul 28 01:17:10.018735 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.018713 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51fdc520-b7b9-45fc-b93f-c220cab87fa9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6465894bc4-lxqgl\" (UID: \"51fdc520-b7b9-45fc-b93f-c220cab87fa9\") " pod="openshift-monitoring/metrics-server-6465894bc4-lxqgl" Jul 28 01:17:10.018876 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.018857 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/51fdc520-b7b9-45fc-b93f-c220cab87fa9-metrics-server-audit-profiles\") pod \"metrics-server-6465894bc4-lxqgl\" (UID: \"51fdc520-b7b9-45fc-b93f-c220cab87fa9\") " pod="openshift-monitoring/metrics-server-6465894bc4-lxqgl" Jul 28 01:17:10.020805 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.020764 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/51fdc520-b7b9-45fc-b93f-c220cab87fa9-client-ca-bundle\") pod \"metrics-server-6465894bc4-lxqgl\" (UID: \"51fdc520-b7b9-45fc-b93f-c220cab87fa9\") " pod="openshift-monitoring/metrics-server-6465894bc4-lxqgl" Jul 28 01:17:10.020923 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.020823 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/51fdc520-b7b9-45fc-b93f-c220cab87fa9-secret-metrics-server-tls\") pod \"metrics-server-6465894bc4-lxqgl\" (UID: \"51fdc520-b7b9-45fc-b93f-c220cab87fa9\") " pod="openshift-monitoring/metrics-server-6465894bc4-lxqgl" Jul 28 01:17:10.021132 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.021110 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/51fdc520-b7b9-45fc-b93f-c220cab87fa9-secret-metrics-server-client-certs\") pod \"metrics-server-6465894bc4-lxqgl\" (UID: \"51fdc520-b7b9-45fc-b93f-c220cab87fa9\") " pod="openshift-monitoring/metrics-server-6465894bc4-lxqgl" Jul 28 01:17:10.025685 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.025663 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2gdw\" (UniqueName: \"kubernetes.io/projected/51fdc520-b7b9-45fc-b93f-c220cab87fa9-kube-api-access-b2gdw\") pod \"metrics-server-6465894bc4-lxqgl\" (UID: \"51fdc520-b7b9-45fc-b93f-c220cab87fa9\") " pod="openshift-monitoring/metrics-server-6465894bc4-lxqgl" Jul 28 01:17:10.141188 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.141163 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6465894bc4-lxqgl" Jul 28 01:17:10.277006 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.276928 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6465894bc4-lxqgl"] Jul 28 01:17:10.280351 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:17:10.280325 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51fdc520_b7b9_45fc_b93f_c220cab87fa9.slice/crio-11c3bb126f92c4033b690e64f3ee25b36324ddb64562f27c97e4cc86a2667794 WatchSource:0}: Error finding container 11c3bb126f92c4033b690e64f3ee25b36324ddb64562f27c97e4cc86a2667794: Status 404 returned error can't find the container with id 11c3bb126f92c4033b690e64f3ee25b36324ddb64562f27c97e4cc86a2667794 Jul 28 01:17:10.481953 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.481919 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6465894bc4-lxqgl" event={"ID":"51fdc520-b7b9-45fc-b93f-c220cab87fa9","Type":"ContainerStarted","Data":"11c3bb126f92c4033b690e64f3ee25b36324ddb64562f27c97e4cc86a2667794"} Jul 28 01:17:10.483420 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.483392 2565 generic.go:356] "Generic (PLEG): container finished" podID="8e290194-0951-4cc1-b02e-9f40fcf53cf0" containerID="a65698339fe1b14f410660bd154f0fcc1a62d76301ef37273b0e0f17ee853650" exitCode=0 Jul 28 01:17:10.483554 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.483476 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e290194-0951-4cc1-b02e-9f40fcf53cf0","Type":"ContainerDied","Data":"a65698339fe1b14f410660bd154f0fcc1a62d76301ef37273b0e0f17ee853650"} Jul 28 01:17:10.485083 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.485059 2565 generic.go:356] "Generic (PLEG): container finished" podID="94413823-494b-477d-9cc9-51f4bcb2e16f" containerID="ac2d0fb2c696a4a05af53510f333b0f3825f5d42aa62c4540066f46051eea820" exitCode=0 Jul 28 01:17:10.485173 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.485131 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qkgnd" event={"ID":"94413823-494b-477d-9cc9-51f4bcb2e16f","Type":"ContainerDied","Data":"ac2d0fb2c696a4a05af53510f333b0f3825f5d42aa62c4540066f46051eea820"} Jul 28 01:17:10.560368 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.560339 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bb77984d7-rq6wz"] Jul 28 01:17:10.598805 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.598770 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bb77984d7-rq6wz"] Jul 28 01:17:10.598967 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.598870 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bb77984d7-rq6wz" Jul 28 01:17:10.723390 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.723345 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82a772af-8dca-4f33-bb6f-ec566dfd2081-trusted-ca-bundle\") pod \"console-7bb77984d7-rq6wz\" (UID: \"82a772af-8dca-4f33-bb6f-ec566dfd2081\") " pod="openshift-console/console-7bb77984d7-rq6wz" Jul 28 01:17:10.723535 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.723408 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82a772af-8dca-4f33-bb6f-ec566dfd2081-console-serving-cert\") pod \"console-7bb77984d7-rq6wz\" (UID: \"82a772af-8dca-4f33-bb6f-ec566dfd2081\") " pod="openshift-console/console-7bb77984d7-rq6wz" Jul 28 01:17:10.723535 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.723462 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82a772af-8dca-4f33-bb6f-ec566dfd2081-oauth-serving-cert\") pod \"console-7bb77984d7-rq6wz\" (UID: \"82a772af-8dca-4f33-bb6f-ec566dfd2081\") " pod="openshift-console/console-7bb77984d7-rq6wz" Jul 28 01:17:10.723535 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.723513 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82a772af-8dca-4f33-bb6f-ec566dfd2081-console-config\") pod \"console-7bb77984d7-rq6wz\" (UID: \"82a772af-8dca-4f33-bb6f-ec566dfd2081\") " pod="openshift-console/console-7bb77984d7-rq6wz" Jul 28 01:17:10.723637 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.723545 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czqcn\" (UniqueName: \"kubernetes.io/projected/82a772af-8dca-4f33-bb6f-ec566dfd2081-kube-api-access-czqcn\") pod \"console-7bb77984d7-rq6wz\" (UID: \"82a772af-8dca-4f33-bb6f-ec566dfd2081\") " pod="openshift-console/console-7bb77984d7-rq6wz" Jul 28 01:17:10.723637 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.723574 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82a772af-8dca-4f33-bb6f-ec566dfd2081-console-oauth-config\") pod \"console-7bb77984d7-rq6wz\" (UID: \"82a772af-8dca-4f33-bb6f-ec566dfd2081\") " pod="openshift-console/console-7bb77984d7-rq6wz" Jul 28 01:17:10.723698 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.723642 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82a772af-8dca-4f33-bb6f-ec566dfd2081-service-ca\") pod \"console-7bb77984d7-rq6wz\" (UID: \"82a772af-8dca-4f33-bb6f-ec566dfd2081\") " pod="openshift-console/console-7bb77984d7-rq6wz" Jul 28 01:17:10.824287 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.824198 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82a772af-8dca-4f33-bb6f-ec566dfd2081-console-config\") pod \"console-7bb77984d7-rq6wz\" (UID: \"82a772af-8dca-4f33-bb6f-ec566dfd2081\") " pod="openshift-console/console-7bb77984d7-rq6wz" Jul 28 01:17:10.824287 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.824251 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czqcn\" (UniqueName: \"kubernetes.io/projected/82a772af-8dca-4f33-bb6f-ec566dfd2081-kube-api-access-czqcn\") pod \"console-7bb77984d7-rq6wz\" (UID: \"82a772af-8dca-4f33-bb6f-ec566dfd2081\") " pod="openshift-console/console-7bb77984d7-rq6wz" Jul 28 01:17:10.824476 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.824287 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82a772af-8dca-4f33-bb6f-ec566dfd2081-console-oauth-config\") pod \"console-7bb77984d7-rq6wz\" (UID: \"82a772af-8dca-4f33-bb6f-ec566dfd2081\") " pod="openshift-console/console-7bb77984d7-rq6wz" Jul 28 01:17:10.824476 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.824314 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82a772af-8dca-4f33-bb6f-ec566dfd2081-service-ca\") pod \"console-7bb77984d7-rq6wz\" (UID: \"82a772af-8dca-4f33-bb6f-ec566dfd2081\") " pod="openshift-console/console-7bb77984d7-rq6wz" Jul 28 01:17:10.824476 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.824442 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82a772af-8dca-4f33-bb6f-ec566dfd2081-trusted-ca-bundle\") pod \"console-7bb77984d7-rq6wz\" (UID: \"82a772af-8dca-4f33-bb6f-ec566dfd2081\") " pod="openshift-console/console-7bb77984d7-rq6wz" Jul 28 01:17:10.824616 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.824508 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82a772af-8dca-4f33-bb6f-ec566dfd2081-console-serving-cert\") pod \"console-7bb77984d7-rq6wz\" (UID: \"82a772af-8dca-4f33-bb6f-ec566dfd2081\") " pod="openshift-console/console-7bb77984d7-rq6wz" Jul 28 01:17:10.824616 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.824544 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82a772af-8dca-4f33-bb6f-ec566dfd2081-oauth-serving-cert\") pod \"console-7bb77984d7-rq6wz\" (UID: \"82a772af-8dca-4f33-bb6f-ec566dfd2081\") " pod="openshift-console/console-7bb77984d7-rq6wz" Jul 28 01:17:10.824999 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.824973 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82a772af-8dca-4f33-bb6f-ec566dfd2081-console-config\") pod \"console-7bb77984d7-rq6wz\" (UID: \"82a772af-8dca-4f33-bb6f-ec566dfd2081\") " pod="openshift-console/console-7bb77984d7-rq6wz" Jul 28 01:17:10.825200 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.825146 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82a772af-8dca-4f33-bb6f-ec566dfd2081-service-ca\") pod \"console-7bb77984d7-rq6wz\" (UID: \"82a772af-8dca-4f33-bb6f-ec566dfd2081\") " pod="openshift-console/console-7bb77984d7-rq6wz" Jul 28 01:17:10.825293 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.825273 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82a772af-8dca-4f33-bb6f-ec566dfd2081-oauth-serving-cert\") pod \"console-7bb77984d7-rq6wz\" (UID: \"82a772af-8dca-4f33-bb6f-ec566dfd2081\") " pod="openshift-console/console-7bb77984d7-rq6wz" Jul 28 01:17:10.825436 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.825417 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82a772af-8dca-4f33-bb6f-ec566dfd2081-trusted-ca-bundle\") pod \"console-7bb77984d7-rq6wz\" (UID: \"82a772af-8dca-4f33-bb6f-ec566dfd2081\") " pod="openshift-console/console-7bb77984d7-rq6wz" Jul 28 01:17:10.826791 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.826767 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82a772af-8dca-4f33-bb6f-ec566dfd2081-console-serving-cert\") pod \"console-7bb77984d7-rq6wz\" (UID: \"82a772af-8dca-4f33-bb6f-ec566dfd2081\") " pod="openshift-console/console-7bb77984d7-rq6wz" Jul 28 01:17:10.826939 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.826922 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82a772af-8dca-4f33-bb6f-ec566dfd2081-console-oauth-config\") pod \"console-7bb77984d7-rq6wz\" (UID: \"82a772af-8dca-4f33-bb6f-ec566dfd2081\") " pod="openshift-console/console-7bb77984d7-rq6wz" Jul 28 01:17:10.838323 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.838302 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czqcn\" (UniqueName: \"kubernetes.io/projected/82a772af-8dca-4f33-bb6f-ec566dfd2081-kube-api-access-czqcn\") pod \"console-7bb77984d7-rq6wz\" (UID: \"82a772af-8dca-4f33-bb6f-ec566dfd2081\") " pod="openshift-console/console-7bb77984d7-rq6wz" Jul 28 01:17:10.909101 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:10.909066 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bb77984d7-rq6wz" Jul 28 01:17:11.059248 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:11.059054 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bb77984d7-rq6wz"] Jul 28 01:17:11.062207 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:17:11.062180 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82a772af_8dca_4f33_bb6f_ec566dfd2081.slice/crio-75a259a56a37b46b20246fd1112d8a1d90a5b4d1c58c777283af1a3606da44c1 WatchSource:0}: Error finding container 75a259a56a37b46b20246fd1112d8a1d90a5b4d1c58c777283af1a3606da44c1: Status 404 returned error can't find the container with id 75a259a56a37b46b20246fd1112d8a1d90a5b4d1c58c777283af1a3606da44c1 Jul 28 01:17:11.493687 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:11.493201 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qkgnd" event={"ID":"94413823-494b-477d-9cc9-51f4bcb2e16f","Type":"ContainerStarted","Data":"221595279f859c686619455e3d70537ec0c0b37e979a4f8b8edc90767aad6ea4"} Jul 28 01:17:11.493687 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:11.493297 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qkgnd" event={"ID":"94413823-494b-477d-9cc9-51f4bcb2e16f","Type":"ContainerStarted","Data":"f66427cc5f858ba63e6c9979a66d87ef4f6a20e7345dfb3ab89d78c851a3f98c"} Jul 28 01:17:11.495504 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:11.495475 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bb77984d7-rq6wz" event={"ID":"82a772af-8dca-4f33-bb6f-ec566dfd2081","Type":"ContainerStarted","Data":"aa714e2dfb4b0d4578d034de1d64ed62ff02ad1c897cc061e2f1f944c7fd00dc"} Jul 28 01:17:11.495640 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:11.495526 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bb77984d7-rq6wz" event={"ID":"82a772af-8dca-4f33-bb6f-ec566dfd2081","Type":"ContainerStarted","Data":"75a259a56a37b46b20246fd1112d8a1d90a5b4d1c58c777283af1a3606da44c1"} Jul 28 01:17:11.510772 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:11.510719 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qkgnd" podStartSLOduration=5.515306152 podStartE2EDuration="6.510698839s" totalImagesPullingTime="922.342556ms" totalInitContainerRuntime="73.050131ms" isStatefulPod=true podCreationTimestamp="2026-07-28 01:17:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:17:11.51041909 +0000 UTC m=+88.944817406" watchObservedRunningTime="2026-07-28 01:17:11.510698839 +0000 UTC m=+88.945097156" Jul 28 01:17:11.527715 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:11.527651 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bb77984d7-rq6wz" podStartSLOduration=1.5276279499999998 podStartE2EDuration="1.52762795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:17:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:17:11.526091243 +0000 UTC m=+88.960489558" watchObservedRunningTime="2026-07-28 01:17:11.52762795 +0000 UTC m=+88.962026269" Jul 28 01:17:12.502452 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:12.502414 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e290194-0951-4cc1-b02e-9f40fcf53cf0","Type":"ContainerStarted","Data":"e04d3f6243d575ade0a0bad14f100b670a0f8cb7901980e2ccdc9da75851a978"} Jul 28 01:17:12.502845 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:12.502553 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e290194-0951-4cc1-b02e-9f40fcf53cf0","Type":"ContainerStarted","Data":"8452ca82f62fc794efb7916e6fbdc4c8a40ce54e6c34c118adc1d26cb1af8d7a"} Jul 28 01:17:12.502845 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:12.502802 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e290194-0951-4cc1-b02e-9f40fcf53cf0","Type":"ContainerStarted","Data":"f897ff2800b97ed58dcd026e41e0d271020f0a3ff27120996cc7e8a440da1fba"} Jul 28 01:17:12.503939 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:12.503861 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6465894bc4-lxqgl" event={"ID":"51fdc520-b7b9-45fc-b93f-c220cab87fa9","Type":"ContainerStarted","Data":"77a0bafb2e01194c6eaa6ff540dcf4d7b4891773bf1f6250b03ffadcf0bcccce"} Jul 28 01:17:12.520709 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:12.520653 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6465894bc4-lxqgl" podStartSLOduration=1.544516032 podStartE2EDuration="3.520633099s" totalImagesPullingTime="1.976117067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:17:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:17:12.519680612 +0000 UTC m=+89.954079121" watchObservedRunningTime="2026-07-28 01:17:12.520633099 +0000 UTC m=+89.955031414" Jul 28 01:17:13.511423 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:13.511386 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e290194-0951-4cc1-b02e-9f40fcf53cf0","Type":"ContainerStarted","Data":"b8bd7ba899dcd6c1360cd73b8222d7896742f7b24c9ed0cee2a4ad463fc38941"} Jul 28 01:17:13.511934 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:13.511450 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e290194-0951-4cc1-b02e-9f40fcf53cf0","Type":"ContainerStarted","Data":"1f0b620a0b494ce7febf57d043be1519a66dcd33a0f16e0cc942804ac597065d"} Jul 28 01:17:13.511934 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:13.511478 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e290194-0951-4cc1-b02e-9f40fcf53cf0","Type":"ContainerStarted","Data":"c89a4b3983400d5340c36ea23dd217fb99d271e4eb9bbbf7e61ac902fa2f4426"} Jul 28 01:17:13.539733 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:13.539684 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.796655741 podStartE2EDuration="7.539665869s" totalImagesPullingTime="3.720467466s" totalInitContainerRuntime="22.542662ms" isStatefulPod=false podCreationTimestamp="2026-07-28 01:17:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:17:13.538293048 +0000 UTC m=+90.972691374" watchObservedRunningTime="2026-07-28 01:17:13.539665869 +0000 UTC m=+90.974064203" Jul 28 01:17:20.909684 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:20.909643 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bb77984d7-rq6wz" Jul 28 01:17:20.910113 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:20.909736 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bb77984d7-rq6wz" Jul 28 01:17:20.914476 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:20.914449 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7bb77984d7-rq6wz" Jul 28 01:17:21.545172 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:21.545143 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7bb77984d7-rq6wz" Jul 28 01:17:21.592850 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:21.592810 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cd974fb57-xvht8"] Jul 28 01:17:22.303864 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:22.303829 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f64cd8867-22rqd"] Jul 28 01:17:22.306549 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:22.306526 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f64cd8867-22rqd" Jul 28 01:17:22.323062 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:22.323019 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81615a60-9e59-4450-b53b-e36377150964-service-ca\") pod \"console-5f64cd8867-22rqd\" (UID: \"81615a60-9e59-4450-b53b-e36377150964\") " pod="openshift-console/console-5f64cd8867-22rqd" Jul 28 01:17:22.323212 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:22.323105 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81615a60-9e59-4450-b53b-e36377150964-console-config\") pod \"console-5f64cd8867-22rqd\" (UID: \"81615a60-9e59-4450-b53b-e36377150964\") " pod="openshift-console/console-5f64cd8867-22rqd" Jul 28 01:17:22.323212 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:22.323132 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81615a60-9e59-4450-b53b-e36377150964-trusted-ca-bundle\") pod \"console-5f64cd8867-22rqd\" (UID: \"81615a60-9e59-4450-b53b-e36377150964\") " pod="openshift-console/console-5f64cd8867-22rqd" Jul 28 01:17:22.323212 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:22.323158 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81615a60-9e59-4450-b53b-e36377150964-oauth-serving-cert\") pod \"console-5f64cd8867-22rqd\" (UID: \"81615a60-9e59-4450-b53b-e36377150964\") " pod="openshift-console/console-5f64cd8867-22rqd" Jul 28 01:17:22.323362 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:22.323223 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvh4z\" (UniqueName: \"kubernetes.io/projected/81615a60-9e59-4450-b53b-e36377150964-kube-api-access-rvh4z\") pod \"console-5f64cd8867-22rqd\" (UID: \"81615a60-9e59-4450-b53b-e36377150964\") " pod="openshift-console/console-5f64cd8867-22rqd" Jul 28 01:17:22.323362 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:22.323294 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81615a60-9e59-4450-b53b-e36377150964-console-serving-cert\") pod \"console-5f64cd8867-22rqd\" (UID: \"81615a60-9e59-4450-b53b-e36377150964\") " pod="openshift-console/console-5f64cd8867-22rqd" Jul 28 01:17:22.323362 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:22.323357 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81615a60-9e59-4450-b53b-e36377150964-console-oauth-config\") pod \"console-5f64cd8867-22rqd\" (UID: \"81615a60-9e59-4450-b53b-e36377150964\") " pod="openshift-console/console-5f64cd8867-22rqd" Jul 28 01:17:22.332119 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:22.332098 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f64cd8867-22rqd"] Jul 28 01:17:22.424585 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:22.424548 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81615a60-9e59-4450-b53b-e36377150964-console-oauth-config\") pod \"console-5f64cd8867-22rqd\" (UID: \"81615a60-9e59-4450-b53b-e36377150964\") " pod="openshift-console/console-5f64cd8867-22rqd" Jul 28 01:17:22.424775 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:22.424594 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81615a60-9e59-4450-b53b-e36377150964-service-ca\") pod \"console-5f64cd8867-22rqd\" (UID: \"81615a60-9e59-4450-b53b-e36377150964\") " pod="openshift-console/console-5f64cd8867-22rqd" Jul 28 01:17:22.424775 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:22.424624 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81615a60-9e59-4450-b53b-e36377150964-console-config\") pod \"console-5f64cd8867-22rqd\" (UID: \"81615a60-9e59-4450-b53b-e36377150964\") " pod="openshift-console/console-5f64cd8867-22rqd" Jul 28 01:17:22.424775 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:22.424644 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81615a60-9e59-4450-b53b-e36377150964-trusted-ca-bundle\") pod \"console-5f64cd8867-22rqd\" (UID: \"81615a60-9e59-4450-b53b-e36377150964\") " pod="openshift-console/console-5f64cd8867-22rqd" Jul 28 01:17:22.424775 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:22.424666 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81615a60-9e59-4450-b53b-e36377150964-oauth-serving-cert\") pod \"console-5f64cd8867-22rqd\" (UID: \"81615a60-9e59-4450-b53b-e36377150964\") " pod="openshift-console/console-5f64cd8867-22rqd" Jul 28 01:17:22.424983 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:22.424836 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvh4z\" (UniqueName: \"kubernetes.io/projected/81615a60-9e59-4450-b53b-e36377150964-kube-api-access-rvh4z\") pod \"console-5f64cd8867-22rqd\" (UID: \"81615a60-9e59-4450-b53b-e36377150964\") " pod="openshift-console/console-5f64cd8867-22rqd" Jul 28 01:17:22.424983 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:22.424898 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81615a60-9e59-4450-b53b-e36377150964-console-serving-cert\") pod \"console-5f64cd8867-22rqd\" (UID: \"81615a60-9e59-4450-b53b-e36377150964\") " pod="openshift-console/console-5f64cd8867-22rqd" Jul 28 01:17:22.425502 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:22.425477 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81615a60-9e59-4450-b53b-e36377150964-service-ca\") pod \"console-5f64cd8867-22rqd\" (UID: \"81615a60-9e59-4450-b53b-e36377150964\") " pod="openshift-console/console-5f64cd8867-22rqd" Jul 28 01:17:22.425502 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:22.425497 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81615a60-9e59-4450-b53b-e36377150964-oauth-serving-cert\") pod \"console-5f64cd8867-22rqd\" (UID: \"81615a60-9e59-4450-b53b-e36377150964\") " pod="openshift-console/console-5f64cd8867-22rqd" Jul 28 01:17:22.425672 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:22.425487 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81615a60-9e59-4450-b53b-e36377150964-console-config\") pod \"console-5f64cd8867-22rqd\" (UID: \"81615a60-9e59-4450-b53b-e36377150964\") " pod="openshift-console/console-5f64cd8867-22rqd" Jul 28 01:17:22.425728 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:22.425698 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81615a60-9e59-4450-b53b-e36377150964-trusted-ca-bundle\") pod \"console-5f64cd8867-22rqd\" (UID: \"81615a60-9e59-4450-b53b-e36377150964\") " pod="openshift-console/console-5f64cd8867-22rqd" Jul 28 01:17:22.427088 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:22.427068 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81615a60-9e59-4450-b53b-e36377150964-console-oauth-config\") pod \"console-5f64cd8867-22rqd\" (UID: \"81615a60-9e59-4450-b53b-e36377150964\") " pod="openshift-console/console-5f64cd8867-22rqd" Jul 28 01:17:22.427252 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:22.427232 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81615a60-9e59-4450-b53b-e36377150964-console-serving-cert\") pod \"console-5f64cd8867-22rqd\" (UID: \"81615a60-9e59-4450-b53b-e36377150964\") " pod="openshift-console/console-5f64cd8867-22rqd" Jul 28 01:17:22.436296 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:22.436274 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvh4z\" (UniqueName: \"kubernetes.io/projected/81615a60-9e59-4450-b53b-e36377150964-kube-api-access-rvh4z\") pod \"console-5f64cd8867-22rqd\" (UID: \"81615a60-9e59-4450-b53b-e36377150964\") " pod="openshift-console/console-5f64cd8867-22rqd" Jul 28 01:17:22.616942 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:22.616848 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f64cd8867-22rqd" Jul 28 01:17:22.750119 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:22.750088 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f64cd8867-22rqd"] Jul 28 01:17:22.753047 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:17:22.752994 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81615a60_9e59_4450_b53b_e36377150964.slice/crio-7f1ee53833c7449adadd487f04cb043b8db347ec623111225b0aa3f2ec6d7f01 WatchSource:0}: Error finding container 7f1ee53833c7449adadd487f04cb043b8db347ec623111225b0aa3f2ec6d7f01: Status 404 returned error can't find the container with id 7f1ee53833c7449adadd487f04cb043b8db347ec623111225b0aa3f2ec6d7f01 Jul 28 01:17:23.550203 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:23.550167 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f64cd8867-22rqd" event={"ID":"81615a60-9e59-4450-b53b-e36377150964","Type":"ContainerStarted","Data":"3358945ec12917d200c27f0c5f7e8aa162edb822e892bd7b06ec3c539fd37872"} Jul 28 01:17:23.550705 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:23.550216 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f64cd8867-22rqd" event={"ID":"81615a60-9e59-4450-b53b-e36377150964","Type":"ContainerStarted","Data":"7f1ee53833c7449adadd487f04cb043b8db347ec623111225b0aa3f2ec6d7f01"} Jul 28 01:17:23.568449 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:23.568398 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f64cd8867-22rqd" podStartSLOduration=1.5683825850000002 podStartE2EDuration="1.568382585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:17:23.566262052 +0000 UTC m=+101.000660368" watchObservedRunningTime="2026-07-28 01:17:23.568382585 +0000 UTC m=+101.002780899" Jul 28 01:17:25.407284 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:25.407252 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6swc2" Jul 28 01:17:30.141573 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:30.141528 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6465894bc4-lxqgl" Jul 28 01:17:30.141989 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:30.141598 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6465894bc4-lxqgl" Jul 28 01:17:32.617602 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:32.617561 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f64cd8867-22rqd" Jul 28 01:17:32.617997 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:32.617651 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f64cd8867-22rqd" Jul 28 01:17:32.622808 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:32.622783 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f64cd8867-22rqd" Jul 28 01:17:33.592785 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:33.592757 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f64cd8867-22rqd" Jul 28 01:17:33.635854 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:33.635814 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bb77984d7-rq6wz"] Jul 28 01:17:46.612957 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:46.612914 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-cd974fb57-xvht8" podUID="78de07a8-46c4-4765-a675-8c8a895637d7" containerName="console" containerID="cri-o://77d7e770878a303250654a7ec9148715801f29ab602ed115894d7cd1e16012a3" gracePeriod=15 Jul 28 01:17:46.861224 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:46.861201 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cd974fb57-xvht8" Jul 28 01:17:46.924763 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:46.924679 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pz8vg\" (UniqueName: \"kubernetes.io/projected/78de07a8-46c4-4765-a675-8c8a895637d7-kube-api-access-pz8vg\") pod \"78de07a8-46c4-4765-a675-8c8a895637d7\" (UID: \"78de07a8-46c4-4765-a675-8c8a895637d7\") " Jul 28 01:17:46.924763 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:46.924715 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78de07a8-46c4-4765-a675-8c8a895637d7-trusted-ca-bundle\") pod \"78de07a8-46c4-4765-a675-8c8a895637d7\" (UID: \"78de07a8-46c4-4765-a675-8c8a895637d7\") " Jul 28 01:17:46.924763 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:46.924743 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78de07a8-46c4-4765-a675-8c8a895637d7-console-oauth-config\") pod \"78de07a8-46c4-4765-a675-8c8a895637d7\" (UID: \"78de07a8-46c4-4765-a675-8c8a895637d7\") " Jul 28 01:17:46.924763 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:46.924761 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78de07a8-46c4-4765-a675-8c8a895637d7-console-config\") pod \"78de07a8-46c4-4765-a675-8c8a895637d7\" (UID: \"78de07a8-46c4-4765-a675-8c8a895637d7\") " Jul 28 01:17:46.925010 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:46.924945 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78de07a8-46c4-4765-a675-8c8a895637d7-oauth-serving-cert\") pod \"78de07a8-46c4-4765-a675-8c8a895637d7\" (UID: \"78de07a8-46c4-4765-a675-8c8a895637d7\") " Jul 28 01:17:46.925010 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:46.924979 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78de07a8-46c4-4765-a675-8c8a895637d7-console-serving-cert\") pod \"78de07a8-46c4-4765-a675-8c8a895637d7\" (UID: \"78de07a8-46c4-4765-a675-8c8a895637d7\") " Jul 28 01:17:46.925010 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:46.925007 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78de07a8-46c4-4765-a675-8c8a895637d7-service-ca\") pod \"78de07a8-46c4-4765-a675-8c8a895637d7\" (UID: \"78de07a8-46c4-4765-a675-8c8a895637d7\") " Jul 28 01:17:46.925338 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:46.925279 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78de07a8-46c4-4765-a675-8c8a895637d7-console-config" (OuterVolumeSpecName: "console-config") pod "78de07a8-46c4-4765-a675-8c8a895637d7" (UID: "78de07a8-46c4-4765-a675-8c8a895637d7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:17:46.925338 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:46.925314 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78de07a8-46c4-4765-a675-8c8a895637d7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "78de07a8-46c4-4765-a675-8c8a895637d7" (UID: "78de07a8-46c4-4765-a675-8c8a895637d7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:17:46.925523 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:46.925480 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78de07a8-46c4-4765-a675-8c8a895637d7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "78de07a8-46c4-4765-a675-8c8a895637d7" (UID: "78de07a8-46c4-4765-a675-8c8a895637d7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:17:46.925523 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:46.925505 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78de07a8-46c4-4765-a675-8c8a895637d7-service-ca" (OuterVolumeSpecName: "service-ca") pod "78de07a8-46c4-4765-a675-8c8a895637d7" (UID: "78de07a8-46c4-4765-a675-8c8a895637d7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:17:46.927027 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:46.927001 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78de07a8-46c4-4765-a675-8c8a895637d7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "78de07a8-46c4-4765-a675-8c8a895637d7" (UID: "78de07a8-46c4-4765-a675-8c8a895637d7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:17:46.927027 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:46.927012 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78de07a8-46c4-4765-a675-8c8a895637d7-kube-api-access-pz8vg" (OuterVolumeSpecName: "kube-api-access-pz8vg") pod "78de07a8-46c4-4765-a675-8c8a895637d7" (UID: "78de07a8-46c4-4765-a675-8c8a895637d7"). InnerVolumeSpecName "kube-api-access-pz8vg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:17:46.927177 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:46.927018 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78de07a8-46c4-4765-a675-8c8a895637d7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "78de07a8-46c4-4765-a675-8c8a895637d7" (UID: "78de07a8-46c4-4765-a675-8c8a895637d7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:17:47.026638 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:47.026596 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78de07a8-46c4-4765-a675-8c8a895637d7-service-ca\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:17:47.026638 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:47.026627 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz8vg\" (UniqueName: \"kubernetes.io/projected/78de07a8-46c4-4765-a675-8c8a895637d7-kube-api-access-pz8vg\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:17:47.026638 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:47.026637 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78de07a8-46c4-4765-a675-8c8a895637d7-trusted-ca-bundle\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:17:47.026638 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:47.026646 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78de07a8-46c4-4765-a675-8c8a895637d7-console-oauth-config\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:17:47.026900 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:47.026655 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78de07a8-46c4-4765-a675-8c8a895637d7-console-config\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:17:47.026900 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:47.026665 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78de07a8-46c4-4765-a675-8c8a895637d7-oauth-serving-cert\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:17:47.026900 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:47.026673 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78de07a8-46c4-4765-a675-8c8a895637d7-console-serving-cert\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:17:47.638808 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:47.638770 2565 generic.go:356] "Generic (PLEG): container finished" podID="78de07a8-46c4-4765-a675-8c8a895637d7" containerID="77d7e770878a303250654a7ec9148715801f29ab602ed115894d7cd1e16012a3" exitCode=0 Jul 28 01:17:47.639231 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:47.638818 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cd974fb57-xvht8" Jul 28 01:17:47.639231 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:47.638860 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cd974fb57-xvht8" event={"ID":"78de07a8-46c4-4765-a675-8c8a895637d7","Type":"ContainerDied","Data":"77d7e770878a303250654a7ec9148715801f29ab602ed115894d7cd1e16012a3"} Jul 28 01:17:47.639231 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:47.638913 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cd974fb57-xvht8" event={"ID":"78de07a8-46c4-4765-a675-8c8a895637d7","Type":"ContainerDied","Data":"2c3c9a4a9cb5b8b471a4cdb103c78d3b51cfc6eefb50f623d1135302a2e9a727"} Jul 28 01:17:47.639231 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:47.638933 2565 scope.go:117] "RemoveContainer" containerID="77d7e770878a303250654a7ec9148715801f29ab602ed115894d7cd1e16012a3" Jul 28 01:17:47.646970 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:47.646953 2565 scope.go:117] "RemoveContainer" containerID="77d7e770878a303250654a7ec9148715801f29ab602ed115894d7cd1e16012a3" Jul 28 01:17:47.647268 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:17:47.647249 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77d7e770878a303250654a7ec9148715801f29ab602ed115894d7cd1e16012a3\": container with ID starting with 77d7e770878a303250654a7ec9148715801f29ab602ed115894d7cd1e16012a3 not found: ID does not exist" containerID="77d7e770878a303250654a7ec9148715801f29ab602ed115894d7cd1e16012a3" Jul 28 01:17:47.647319 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:47.647275 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77d7e770878a303250654a7ec9148715801f29ab602ed115894d7cd1e16012a3"} err="failed to get container status \"77d7e770878a303250654a7ec9148715801f29ab602ed115894d7cd1e16012a3\": rpc error: code = NotFound desc = could not find container \"77d7e770878a303250654a7ec9148715801f29ab602ed115894d7cd1e16012a3\": container with ID starting with 77d7e770878a303250654a7ec9148715801f29ab602ed115894d7cd1e16012a3 not found: ID does not exist" Jul 28 01:17:47.655384 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:47.655335 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cd974fb57-xvht8"] Jul 28 01:17:47.660597 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:47.657835 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cd974fb57-xvht8"] Jul 28 01:17:49.100543 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:49.100507 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78de07a8-46c4-4765-a675-8c8a895637d7" path="/var/lib/kubelet/pods/78de07a8-46c4-4765-a675-8c8a895637d7/volumes" Jul 28 01:17:50.147200 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:50.147168 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6465894bc4-lxqgl" Jul 28 01:17:50.151091 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:50.151064 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6465894bc4-lxqgl" Jul 28 01:17:58.655571 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:58.655535 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7bb77984d7-rq6wz" podUID="82a772af-8dca-4f33-bb6f-ec566dfd2081" containerName="console" containerID="cri-o://aa714e2dfb4b0d4578d034de1d64ed62ff02ad1c897cc061e2f1f944c7fd00dc" gracePeriod=15 Jul 28 01:17:58.905494 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:58.905470 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bb77984d7-rq6wz" Jul 28 01:17:59.025381 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:59.025328 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82a772af-8dca-4f33-bb6f-ec566dfd2081-console-serving-cert\") pod \"82a772af-8dca-4f33-bb6f-ec566dfd2081\" (UID: \"82a772af-8dca-4f33-bb6f-ec566dfd2081\") " Jul 28 01:17:59.025381 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:59.025384 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82a772af-8dca-4f33-bb6f-ec566dfd2081-console-config\") pod \"82a772af-8dca-4f33-bb6f-ec566dfd2081\" (UID: \"82a772af-8dca-4f33-bb6f-ec566dfd2081\") " Jul 28 01:17:59.025700 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:59.025425 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82a772af-8dca-4f33-bb6f-ec566dfd2081-service-ca\") pod \"82a772af-8dca-4f33-bb6f-ec566dfd2081\" (UID: \"82a772af-8dca-4f33-bb6f-ec566dfd2081\") " Jul 28 01:17:59.025700 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:59.025453 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82a772af-8dca-4f33-bb6f-ec566dfd2081-trusted-ca-bundle\") pod \"82a772af-8dca-4f33-bb6f-ec566dfd2081\" (UID: \"82a772af-8dca-4f33-bb6f-ec566dfd2081\") " Jul 28 01:17:59.025700 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:59.025503 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82a772af-8dca-4f33-bb6f-ec566dfd2081-oauth-serving-cert\") pod \"82a772af-8dca-4f33-bb6f-ec566dfd2081\" (UID: \"82a772af-8dca-4f33-bb6f-ec566dfd2081\") " Jul 28 01:17:59.025700 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:59.025538 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czqcn\" (UniqueName: \"kubernetes.io/projected/82a772af-8dca-4f33-bb6f-ec566dfd2081-kube-api-access-czqcn\") pod \"82a772af-8dca-4f33-bb6f-ec566dfd2081\" (UID: \"82a772af-8dca-4f33-bb6f-ec566dfd2081\") " Jul 28 01:17:59.025700 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:59.025579 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82a772af-8dca-4f33-bb6f-ec566dfd2081-console-oauth-config\") pod \"82a772af-8dca-4f33-bb6f-ec566dfd2081\" (UID: \"82a772af-8dca-4f33-bb6f-ec566dfd2081\") " Jul 28 01:17:59.025952 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:59.025899 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82a772af-8dca-4f33-bb6f-ec566dfd2081-service-ca" (OuterVolumeSpecName: "service-ca") pod "82a772af-8dca-4f33-bb6f-ec566dfd2081" (UID: "82a772af-8dca-4f33-bb6f-ec566dfd2081"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:17:59.025952 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:59.025910 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82a772af-8dca-4f33-bb6f-ec566dfd2081-console-config" (OuterVolumeSpecName: "console-config") pod "82a772af-8dca-4f33-bb6f-ec566dfd2081" (UID: "82a772af-8dca-4f33-bb6f-ec566dfd2081"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:17:59.026137 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:59.025945 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82a772af-8dca-4f33-bb6f-ec566dfd2081-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "82a772af-8dca-4f33-bb6f-ec566dfd2081" (UID: "82a772af-8dca-4f33-bb6f-ec566dfd2081"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:17:59.026137 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:59.025966 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82a772af-8dca-4f33-bb6f-ec566dfd2081-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "82a772af-8dca-4f33-bb6f-ec566dfd2081" (UID: "82a772af-8dca-4f33-bb6f-ec566dfd2081"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:17:59.027654 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:59.027631 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82a772af-8dca-4f33-bb6f-ec566dfd2081-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "82a772af-8dca-4f33-bb6f-ec566dfd2081" (UID: "82a772af-8dca-4f33-bb6f-ec566dfd2081"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:17:59.027749 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:59.027678 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82a772af-8dca-4f33-bb6f-ec566dfd2081-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "82a772af-8dca-4f33-bb6f-ec566dfd2081" (UID: "82a772af-8dca-4f33-bb6f-ec566dfd2081"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:17:59.027749 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:59.027689 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82a772af-8dca-4f33-bb6f-ec566dfd2081-kube-api-access-czqcn" (OuterVolumeSpecName: "kube-api-access-czqcn") pod "82a772af-8dca-4f33-bb6f-ec566dfd2081" (UID: "82a772af-8dca-4f33-bb6f-ec566dfd2081"). InnerVolumeSpecName "kube-api-access-czqcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:17:59.126725 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:59.126700 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82a772af-8dca-4f33-bb6f-ec566dfd2081-service-ca\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:17:59.126725 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:59.126724 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82a772af-8dca-4f33-bb6f-ec566dfd2081-trusted-ca-bundle\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:17:59.126889 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:59.126734 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82a772af-8dca-4f33-bb6f-ec566dfd2081-oauth-serving-cert\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:17:59.126889 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:59.126745 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czqcn\" (UniqueName: \"kubernetes.io/projected/82a772af-8dca-4f33-bb6f-ec566dfd2081-kube-api-access-czqcn\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:17:59.126889 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:59.126753 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82a772af-8dca-4f33-bb6f-ec566dfd2081-console-oauth-config\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:17:59.126889 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:59.126761 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82a772af-8dca-4f33-bb6f-ec566dfd2081-console-serving-cert\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:17:59.126889 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:59.126771 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82a772af-8dca-4f33-bb6f-ec566dfd2081-console-config\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:17:59.683742 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:59.683709 2565 generic.go:356] "Generic (PLEG): container finished" podID="82a772af-8dca-4f33-bb6f-ec566dfd2081" containerID="aa714e2dfb4b0d4578d034de1d64ed62ff02ad1c897cc061e2f1f944c7fd00dc" exitCode=0 Jul 28 01:17:59.684169 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:59.683749 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bb77984d7-rq6wz" Jul 28 01:17:59.684169 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:59.683794 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bb77984d7-rq6wz" event={"ID":"82a772af-8dca-4f33-bb6f-ec566dfd2081","Type":"ContainerDied","Data":"aa714e2dfb4b0d4578d034de1d64ed62ff02ad1c897cc061e2f1f944c7fd00dc"} Jul 28 01:17:59.684169 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:59.683844 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bb77984d7-rq6wz" event={"ID":"82a772af-8dca-4f33-bb6f-ec566dfd2081","Type":"ContainerDied","Data":"75a259a56a37b46b20246fd1112d8a1d90a5b4d1c58c777283af1a3606da44c1"} Jul 28 01:17:59.684169 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:59.683864 2565 scope.go:117] "RemoveContainer" containerID="aa714e2dfb4b0d4578d034de1d64ed62ff02ad1c897cc061e2f1f944c7fd00dc" Jul 28 01:17:59.691427 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:59.691407 2565 scope.go:117] "RemoveContainer" containerID="aa714e2dfb4b0d4578d034de1d64ed62ff02ad1c897cc061e2f1f944c7fd00dc" Jul 28 01:17:59.691679 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:17:59.691654 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa714e2dfb4b0d4578d034de1d64ed62ff02ad1c897cc061e2f1f944c7fd00dc\": container with ID starting with aa714e2dfb4b0d4578d034de1d64ed62ff02ad1c897cc061e2f1f944c7fd00dc not found: ID does not exist" containerID="aa714e2dfb4b0d4578d034de1d64ed62ff02ad1c897cc061e2f1f944c7fd00dc" Jul 28 01:17:59.691725 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:59.691687 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa714e2dfb4b0d4578d034de1d64ed62ff02ad1c897cc061e2f1f944c7fd00dc"} err="failed to get container status \"aa714e2dfb4b0d4578d034de1d64ed62ff02ad1c897cc061e2f1f944c7fd00dc\": rpc error: code = NotFound desc = could not find container \"aa714e2dfb4b0d4578d034de1d64ed62ff02ad1c897cc061e2f1f944c7fd00dc\": container with ID starting with aa714e2dfb4b0d4578d034de1d64ed62ff02ad1c897cc061e2f1f944c7fd00dc not found: ID does not exist" Jul 28 01:17:59.699521 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:59.699502 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bb77984d7-rq6wz"] Jul 28 01:17:59.704010 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:17:59.703986 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bb77984d7-rq6wz"] Jul 28 01:18:01.100052 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:01.100015 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82a772af-8dca-4f33-bb6f-ec566dfd2081" path="/var/lib/kubelet/pods/82a772af-8dca-4f33-bb6f-ec566dfd2081/volumes" Jul 28 01:18:23.270850 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:23.270820 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-58df9db6f4-kxmqm"] Jul 28 01:18:23.272117 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:23.271639 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78de07a8-46c4-4765-a675-8c8a895637d7" containerName="console" Jul 28 01:18:23.272117 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:23.271665 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="78de07a8-46c4-4765-a675-8c8a895637d7" containerName="console" Jul 28 01:18:23.272117 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:23.271689 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82a772af-8dca-4f33-bb6f-ec566dfd2081" containerName="console" Jul 28 01:18:23.272117 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:23.271885 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="82a772af-8dca-4f33-bb6f-ec566dfd2081" containerName="console" Jul 28 01:18:23.272117 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:23.272092 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="82a772af-8dca-4f33-bb6f-ec566dfd2081" containerName="console" Jul 28 01:18:23.272117 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:23.272117 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="78de07a8-46c4-4765-a675-8c8a895637d7" containerName="console" Jul 28 01:18:23.279172 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:23.279148 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58df9db6f4-kxmqm" Jul 28 01:18:23.282820 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:23.282797 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58df9db6f4-kxmqm"] Jul 28 01:18:23.308766 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:23.308740 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/420dc905-c3d6-4b00-b0dd-3e76542c1801-console-serving-cert\") pod \"console-58df9db6f4-kxmqm\" (UID: \"420dc905-c3d6-4b00-b0dd-3e76542c1801\") " pod="openshift-console/console-58df9db6f4-kxmqm" Jul 28 01:18:23.308766 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:23.308767 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/420dc905-c3d6-4b00-b0dd-3e76542c1801-trusted-ca-bundle\") pod \"console-58df9db6f4-kxmqm\" (UID: \"420dc905-c3d6-4b00-b0dd-3e76542c1801\") " pod="openshift-console/console-58df9db6f4-kxmqm" Jul 28 01:18:23.308907 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:23.308794 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98jvw\" (UniqueName: \"kubernetes.io/projected/420dc905-c3d6-4b00-b0dd-3e76542c1801-kube-api-access-98jvw\") pod \"console-58df9db6f4-kxmqm\" (UID: \"420dc905-c3d6-4b00-b0dd-3e76542c1801\") " pod="openshift-console/console-58df9db6f4-kxmqm" Jul 28 01:18:23.308907 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:23.308819 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/420dc905-c3d6-4b00-b0dd-3e76542c1801-oauth-serving-cert\") pod \"console-58df9db6f4-kxmqm\" (UID: \"420dc905-c3d6-4b00-b0dd-3e76542c1801\") " pod="openshift-console/console-58df9db6f4-kxmqm" Jul 28 01:18:23.308907 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:23.308856 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/420dc905-c3d6-4b00-b0dd-3e76542c1801-service-ca\") pod \"console-58df9db6f4-kxmqm\" (UID: \"420dc905-c3d6-4b00-b0dd-3e76542c1801\") " pod="openshift-console/console-58df9db6f4-kxmqm" Jul 28 01:18:23.308907 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:23.308881 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/420dc905-c3d6-4b00-b0dd-3e76542c1801-console-oauth-config\") pod \"console-58df9db6f4-kxmqm\" (UID: \"420dc905-c3d6-4b00-b0dd-3e76542c1801\") " pod="openshift-console/console-58df9db6f4-kxmqm" Jul 28 01:18:23.309053 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:23.308912 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/420dc905-c3d6-4b00-b0dd-3e76542c1801-console-config\") pod \"console-58df9db6f4-kxmqm\" (UID: \"420dc905-c3d6-4b00-b0dd-3e76542c1801\") " pod="openshift-console/console-58df9db6f4-kxmqm" Jul 28 01:18:23.410021 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:23.409985 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/420dc905-c3d6-4b00-b0dd-3e76542c1801-console-serving-cert\") pod \"console-58df9db6f4-kxmqm\" (UID: \"420dc905-c3d6-4b00-b0dd-3e76542c1801\") " pod="openshift-console/console-58df9db6f4-kxmqm" Jul 28 01:18:23.410021 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:23.410020 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/420dc905-c3d6-4b00-b0dd-3e76542c1801-trusted-ca-bundle\") pod \"console-58df9db6f4-kxmqm\" (UID: \"420dc905-c3d6-4b00-b0dd-3e76542c1801\") " pod="openshift-console/console-58df9db6f4-kxmqm" Jul 28 01:18:23.410272 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:23.410057 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98jvw\" (UniqueName: \"kubernetes.io/projected/420dc905-c3d6-4b00-b0dd-3e76542c1801-kube-api-access-98jvw\") pod \"console-58df9db6f4-kxmqm\" (UID: \"420dc905-c3d6-4b00-b0dd-3e76542c1801\") " pod="openshift-console/console-58df9db6f4-kxmqm" Jul 28 01:18:23.410272 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:23.410187 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/420dc905-c3d6-4b00-b0dd-3e76542c1801-oauth-serving-cert\") pod \"console-58df9db6f4-kxmqm\" (UID: \"420dc905-c3d6-4b00-b0dd-3e76542c1801\") " pod="openshift-console/console-58df9db6f4-kxmqm" Jul 28 01:18:23.410272 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:23.410254 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/420dc905-c3d6-4b00-b0dd-3e76542c1801-service-ca\") pod \"console-58df9db6f4-kxmqm\" (UID: \"420dc905-c3d6-4b00-b0dd-3e76542c1801\") " pod="openshift-console/console-58df9db6f4-kxmqm" Jul 28 01:18:23.410422 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:23.410302 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/420dc905-c3d6-4b00-b0dd-3e76542c1801-console-oauth-config\") pod \"console-58df9db6f4-kxmqm\" (UID: \"420dc905-c3d6-4b00-b0dd-3e76542c1801\") " pod="openshift-console/console-58df9db6f4-kxmqm" Jul 28 01:18:23.410422 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:23.410340 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/420dc905-c3d6-4b00-b0dd-3e76542c1801-console-config\") pod \"console-58df9db6f4-kxmqm\" (UID: \"420dc905-c3d6-4b00-b0dd-3e76542c1801\") " pod="openshift-console/console-58df9db6f4-kxmqm" Jul 28 01:18:23.410950 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:23.410920 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/420dc905-c3d6-4b00-b0dd-3e76542c1801-oauth-serving-cert\") pod \"console-58df9db6f4-kxmqm\" (UID: \"420dc905-c3d6-4b00-b0dd-3e76542c1801\") " pod="openshift-console/console-58df9db6f4-kxmqm" Jul 28 01:18:23.411091 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:23.410965 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/420dc905-c3d6-4b00-b0dd-3e76542c1801-service-ca\") pod \"console-58df9db6f4-kxmqm\" (UID: \"420dc905-c3d6-4b00-b0dd-3e76542c1801\") " pod="openshift-console/console-58df9db6f4-kxmqm" Jul 28 01:18:23.411091 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:23.411002 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/420dc905-c3d6-4b00-b0dd-3e76542c1801-console-config\") pod \"console-58df9db6f4-kxmqm\" (UID: \"420dc905-c3d6-4b00-b0dd-3e76542c1801\") " pod="openshift-console/console-58df9db6f4-kxmqm" Jul 28 01:18:23.411172 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:23.411102 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/420dc905-c3d6-4b00-b0dd-3e76542c1801-trusted-ca-bundle\") pod \"console-58df9db6f4-kxmqm\" (UID: \"420dc905-c3d6-4b00-b0dd-3e76542c1801\") " pod="openshift-console/console-58df9db6f4-kxmqm" Jul 28 01:18:23.412613 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:23.412594 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/420dc905-c3d6-4b00-b0dd-3e76542c1801-console-serving-cert\") pod \"console-58df9db6f4-kxmqm\" (UID: \"420dc905-c3d6-4b00-b0dd-3e76542c1801\") " pod="openshift-console/console-58df9db6f4-kxmqm" Jul 28 01:18:23.412613 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:23.412607 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/420dc905-c3d6-4b00-b0dd-3e76542c1801-console-oauth-config\") pod \"console-58df9db6f4-kxmqm\" (UID: \"420dc905-c3d6-4b00-b0dd-3e76542c1801\") " pod="openshift-console/console-58df9db6f4-kxmqm" Jul 28 01:18:23.417806 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:23.417782 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98jvw\" (UniqueName: \"kubernetes.io/projected/420dc905-c3d6-4b00-b0dd-3e76542c1801-kube-api-access-98jvw\") pod \"console-58df9db6f4-kxmqm\" (UID: \"420dc905-c3d6-4b00-b0dd-3e76542c1801\") " pod="openshift-console/console-58df9db6f4-kxmqm" Jul 28 01:18:23.590399 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:23.590322 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58df9db6f4-kxmqm" Jul 28 01:18:23.717401 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:23.717365 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58df9db6f4-kxmqm"] Jul 28 01:18:23.720259 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:18:23.720232 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420dc905_c3d6_4b00_b0dd_3e76542c1801.slice/crio-e0aa16ae9c3d5f81f4757b88173aef6d6e95171203923466c1cc16f63f1b12a7 WatchSource:0}: Error finding container e0aa16ae9c3d5f81f4757b88173aef6d6e95171203923466c1cc16f63f1b12a7: Status 404 returned error can't find the container with id e0aa16ae9c3d5f81f4757b88173aef6d6e95171203923466c1cc16f63f1b12a7 Jul 28 01:18:23.770226 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:23.770128 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58df9db6f4-kxmqm" event={"ID":"420dc905-c3d6-4b00-b0dd-3e76542c1801","Type":"ContainerStarted","Data":"e0aa16ae9c3d5f81f4757b88173aef6d6e95171203923466c1cc16f63f1b12a7"} Jul 28 01:18:24.775487 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:24.775443 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58df9db6f4-kxmqm" event={"ID":"420dc905-c3d6-4b00-b0dd-3e76542c1801","Type":"ContainerStarted","Data":"6a04716f99de47c22006ab21ce2b67d149f1c037b0e42416ba880e1327477e2b"} Jul 28 01:18:24.791727 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:24.791682 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-58df9db6f4-kxmqm" podStartSLOduration=1.791667133 podStartE2EDuration="1.791667133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:18:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:18:24.791461436 +0000 UTC m=+162.225859749" watchObservedRunningTime="2026-07-28 01:18:24.791667133 +0000 UTC m=+162.226065444" Jul 28 01:18:25.879897 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:25.879863 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 01:18:25.880306 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:25.880167 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8e290194-0951-4cc1-b02e-9f40fcf53cf0" containerName="alertmanager" containerID="cri-o://f897ff2800b97ed58dcd026e41e0d271020f0a3ff27120996cc7e8a440da1fba" gracePeriod=120 Jul 28 01:18:25.880306 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:25.880230 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8e290194-0951-4cc1-b02e-9f40fcf53cf0" containerName="kube-rbac-proxy-metric" containerID="cri-o://1f0b620a0b494ce7febf57d043be1519a66dcd33a0f16e0cc942804ac597065d" gracePeriod=120 Jul 28 01:18:25.880306 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:25.880261 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8e290194-0951-4cc1-b02e-9f40fcf53cf0" containerName="kube-rbac-proxy-web" containerID="cri-o://e04d3f6243d575ade0a0bad14f100b670a0f8cb7901980e2ccdc9da75851a978" gracePeriod=120 Jul 28 01:18:25.880306 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:25.880290 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8e290194-0951-4cc1-b02e-9f40fcf53cf0" containerName="config-reloader" containerID="cri-o://8452ca82f62fc794efb7916e6fbdc4c8a40ce54e6c34c118adc1d26cb1af8d7a" gracePeriod=120 Jul 28 01:18:25.880562 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:25.880251 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8e290194-0951-4cc1-b02e-9f40fcf53cf0" containerName="prom-label-proxy" containerID="cri-o://b8bd7ba899dcd6c1360cd73b8222d7896742f7b24c9ed0cee2a4ad463fc38941" gracePeriod=120 Jul 28 01:18:25.880562 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:25.880358 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8e290194-0951-4cc1-b02e-9f40fcf53cf0" containerName="kube-rbac-proxy" containerID="cri-o://c89a4b3983400d5340c36ea23dd217fb99d271e4eb9bbbf7e61ac902fa2f4426" gracePeriod=120 Jul 28 01:18:26.785722 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:26.785689 2565 generic.go:356] "Generic (PLEG): container finished" podID="8e290194-0951-4cc1-b02e-9f40fcf53cf0" containerID="b8bd7ba899dcd6c1360cd73b8222d7896742f7b24c9ed0cee2a4ad463fc38941" exitCode=0 Jul 28 01:18:26.785722 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:26.785715 2565 generic.go:356] "Generic (PLEG): container finished" podID="8e290194-0951-4cc1-b02e-9f40fcf53cf0" containerID="c89a4b3983400d5340c36ea23dd217fb99d271e4eb9bbbf7e61ac902fa2f4426" exitCode=0 Jul 28 01:18:26.785722 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:26.785724 2565 generic.go:356] "Generic (PLEG): container finished" podID="8e290194-0951-4cc1-b02e-9f40fcf53cf0" containerID="e04d3f6243d575ade0a0bad14f100b670a0f8cb7901980e2ccdc9da75851a978" exitCode=0 Jul 28 01:18:26.785722 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:26.785729 2565 generic.go:356] "Generic (PLEG): container finished" podID="8e290194-0951-4cc1-b02e-9f40fcf53cf0" containerID="8452ca82f62fc794efb7916e6fbdc4c8a40ce54e6c34c118adc1d26cb1af8d7a" exitCode=0 Jul 28 01:18:26.785980 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:26.785735 2565 generic.go:356] "Generic (PLEG): container finished" podID="8e290194-0951-4cc1-b02e-9f40fcf53cf0" containerID="f897ff2800b97ed58dcd026e41e0d271020f0a3ff27120996cc7e8a440da1fba" exitCode=0 Jul 28 01:18:26.785980 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:26.785758 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e290194-0951-4cc1-b02e-9f40fcf53cf0","Type":"ContainerDied","Data":"b8bd7ba899dcd6c1360cd73b8222d7896742f7b24c9ed0cee2a4ad463fc38941"} Jul 28 01:18:26.785980 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:26.785822 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e290194-0951-4cc1-b02e-9f40fcf53cf0","Type":"ContainerDied","Data":"c89a4b3983400d5340c36ea23dd217fb99d271e4eb9bbbf7e61ac902fa2f4426"} Jul 28 01:18:26.785980 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:26.785848 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e290194-0951-4cc1-b02e-9f40fcf53cf0","Type":"ContainerDied","Data":"e04d3f6243d575ade0a0bad14f100b670a0f8cb7901980e2ccdc9da75851a978"} Jul 28 01:18:26.785980 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:26.785874 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e290194-0951-4cc1-b02e-9f40fcf53cf0","Type":"ContainerDied","Data":"8452ca82f62fc794efb7916e6fbdc4c8a40ce54e6c34c118adc1d26cb1af8d7a"} Jul 28 01:18:26.785980 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:26.785898 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e290194-0951-4cc1-b02e-9f40fcf53cf0","Type":"ContainerDied","Data":"f897ff2800b97ed58dcd026e41e0d271020f0a3ff27120996cc7e8a440da1fba"} Jul 28 01:18:27.130140 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.130112 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:27.135870 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.135846 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " Jul 28 01:18:27.136056 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.135896 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e290194-0951-4cc1-b02e-9f40fcf53cf0-config-out\") pod \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " Jul 28 01:18:27.136056 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.135926 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e290194-0951-4cc1-b02e-9f40fcf53cf0-tls-assets\") pod \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " Jul 28 01:18:27.136056 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.135963 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-cluster-tls-config\") pod \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " Jul 28 01:18:27.136056 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.135988 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e290194-0951-4cc1-b02e-9f40fcf53cf0-metrics-client-ca\") pod \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " Jul 28 01:18:27.136268 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.136120 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-secret-alertmanager-main-tls\") pod \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " Jul 28 01:18:27.136268 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.136154 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-secret-alertmanager-kube-rbac-proxy\") pod \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " Jul 28 01:18:27.136268 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.136197 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88qmm\" (UniqueName: \"kubernetes.io/projected/8e290194-0951-4cc1-b02e-9f40fcf53cf0-kube-api-access-88qmm\") pod \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " Jul 28 01:18:27.136268 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.136231 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-web-config\") pod \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " Jul 28 01:18:27.136468 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.136270 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-config-volume\") pod \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " Jul 28 01:18:27.136468 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.136307 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-secret-alertmanager-kube-rbac-proxy-web\") pod \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " Jul 28 01:18:27.136468 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.136335 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8e290194-0951-4cc1-b02e-9f40fcf53cf0-alertmanager-main-db\") pod \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " Jul 28 01:18:27.136468 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.136370 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e290194-0951-4cc1-b02e-9f40fcf53cf0-alertmanager-trusted-ca-bundle\") pod \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\" (UID: \"8e290194-0951-4cc1-b02e-9f40fcf53cf0\") " Jul 28 01:18:27.136468 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.136425 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e290194-0951-4cc1-b02e-9f40fcf53cf0-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "8e290194-0951-4cc1-b02e-9f40fcf53cf0" (UID: "8e290194-0951-4cc1-b02e-9f40fcf53cf0"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:18:27.136694 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.136619 2565 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e290194-0951-4cc1-b02e-9f40fcf53cf0-metrics-client-ca\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:18:27.137089 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.137065 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e290194-0951-4cc1-b02e-9f40fcf53cf0-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "8e290194-0951-4cc1-b02e-9f40fcf53cf0" (UID: "8e290194-0951-4cc1-b02e-9f40fcf53cf0"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:18:27.137586 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.137559 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e290194-0951-4cc1-b02e-9f40fcf53cf0-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "8e290194-0951-4cc1-b02e-9f40fcf53cf0" (UID: "8e290194-0951-4cc1-b02e-9f40fcf53cf0"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:18:27.138737 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.138709 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e290194-0951-4cc1-b02e-9f40fcf53cf0-config-out" (OuterVolumeSpecName: "config-out") pod "8e290194-0951-4cc1-b02e-9f40fcf53cf0" (UID: "8e290194-0951-4cc1-b02e-9f40fcf53cf0"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:18:27.138831 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.138790 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e290194-0951-4cc1-b02e-9f40fcf53cf0-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8e290194-0951-4cc1-b02e-9f40fcf53cf0" (UID: "8e290194-0951-4cc1-b02e-9f40fcf53cf0"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:18:27.139207 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.139185 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "8e290194-0951-4cc1-b02e-9f40fcf53cf0" (UID: "8e290194-0951-4cc1-b02e-9f40fcf53cf0"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:18:27.139315 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.139232 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "8e290194-0951-4cc1-b02e-9f40fcf53cf0" (UID: "8e290194-0951-4cc1-b02e-9f40fcf53cf0"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:18:27.139374 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.139332 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-config-volume" (OuterVolumeSpecName: "config-volume") pod "8e290194-0951-4cc1-b02e-9f40fcf53cf0" (UID: "8e290194-0951-4cc1-b02e-9f40fcf53cf0"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:18:27.140405 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.140381 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "8e290194-0951-4cc1-b02e-9f40fcf53cf0" (UID: "8e290194-0951-4cc1-b02e-9f40fcf53cf0"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:18:27.140637 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.140618 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "8e290194-0951-4cc1-b02e-9f40fcf53cf0" (UID: "8e290194-0951-4cc1-b02e-9f40fcf53cf0"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:18:27.140814 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.140801 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e290194-0951-4cc1-b02e-9f40fcf53cf0-kube-api-access-88qmm" (OuterVolumeSpecName: "kube-api-access-88qmm") pod "8e290194-0951-4cc1-b02e-9f40fcf53cf0" (UID: "8e290194-0951-4cc1-b02e-9f40fcf53cf0"). InnerVolumeSpecName "kube-api-access-88qmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:18:27.142931 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.142907 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "8e290194-0951-4cc1-b02e-9f40fcf53cf0" (UID: "8e290194-0951-4cc1-b02e-9f40fcf53cf0"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:18:27.149133 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.149113 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-web-config" (OuterVolumeSpecName: "web-config") pod "8e290194-0951-4cc1-b02e-9f40fcf53cf0" (UID: "8e290194-0951-4cc1-b02e-9f40fcf53cf0"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:18:27.237024 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.236995 2565 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e290194-0951-4cc1-b02e-9f40fcf53cf0-config-out\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:18:27.237024 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.237017 2565 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e290194-0951-4cc1-b02e-9f40fcf53cf0-tls-assets\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:18:27.237024 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.237026 2565 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-cluster-tls-config\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:18:27.237258 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.237057 2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-secret-alertmanager-main-tls\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:18:27.237258 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.237069 2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:18:27.237258 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.237080 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88qmm\" (UniqueName: \"kubernetes.io/projected/8e290194-0951-4cc1-b02e-9f40fcf53cf0-kube-api-access-88qmm\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:18:27.237258 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.237088 2565 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-web-config\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:18:27.237258 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.237096 2565 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-config-volume\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:18:27.237258 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.237104 2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:18:27.237258 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.237112 2565 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8e290194-0951-4cc1-b02e-9f40fcf53cf0-alertmanager-main-db\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:18:27.237258 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.237122 2565 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e290194-0951-4cc1-b02e-9f40fcf53cf0-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:18:27.237258 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.237130 2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8e290194-0951-4cc1-b02e-9f40fcf53cf0-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:18:27.792090 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.792047 2565 generic.go:356] "Generic (PLEG): container finished" podID="8e290194-0951-4cc1-b02e-9f40fcf53cf0" containerID="1f0b620a0b494ce7febf57d043be1519a66dcd33a0f16e0cc942804ac597065d" exitCode=0 Jul 28 01:18:27.792231 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.792125 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:27.792231 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.792128 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e290194-0951-4cc1-b02e-9f40fcf53cf0","Type":"ContainerDied","Data":"1f0b620a0b494ce7febf57d043be1519a66dcd33a0f16e0cc942804ac597065d"} Jul 28 01:18:27.792231 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.792196 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e290194-0951-4cc1-b02e-9f40fcf53cf0","Type":"ContainerDied","Data":"5bc8eab696d5799746c79a9a453d8af6deaf0d63416726c71faf3d840c1e4600"} Jul 28 01:18:27.792231 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.792230 2565 scope.go:117] "RemoveContainer" containerID="b8bd7ba899dcd6c1360cd73b8222d7896742f7b24c9ed0cee2a4ad463fc38941" Jul 28 01:18:27.800243 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.800223 2565 scope.go:117] "RemoveContainer" containerID="1f0b620a0b494ce7febf57d043be1519a66dcd33a0f16e0cc942804ac597065d" Jul 28 01:18:27.807472 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.807451 2565 scope.go:117] "RemoveContainer" containerID="c89a4b3983400d5340c36ea23dd217fb99d271e4eb9bbbf7e61ac902fa2f4426" Jul 28 01:18:27.813596 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.813576 2565 scope.go:117] "RemoveContainer" containerID="e04d3f6243d575ade0a0bad14f100b670a0f8cb7901980e2ccdc9da75851a978" Jul 28 01:18:27.816264 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.816243 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 01:18:27.820071 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.820050 2565 scope.go:117] "RemoveContainer" containerID="8452ca82f62fc794efb7916e6fbdc4c8a40ce54e6c34c118adc1d26cb1af8d7a" Jul 28 01:18:27.825189 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.825166 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 01:18:27.827131 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.827115 2565 scope.go:117] "RemoveContainer" containerID="f897ff2800b97ed58dcd026e41e0d271020f0a3ff27120996cc7e8a440da1fba" Jul 28 01:18:27.833655 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.833638 2565 scope.go:117] "RemoveContainer" containerID="a65698339fe1b14f410660bd154f0fcc1a62d76301ef37273b0e0f17ee853650" Jul 28 01:18:27.842674 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.842658 2565 scope.go:117] "RemoveContainer" containerID="b8bd7ba899dcd6c1360cd73b8222d7896742f7b24c9ed0cee2a4ad463fc38941" Jul 28 01:18:27.842939 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:18:27.842919 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8bd7ba899dcd6c1360cd73b8222d7896742f7b24c9ed0cee2a4ad463fc38941\": container with ID starting with b8bd7ba899dcd6c1360cd73b8222d7896742f7b24c9ed0cee2a4ad463fc38941 not found: ID does not exist" containerID="b8bd7ba899dcd6c1360cd73b8222d7896742f7b24c9ed0cee2a4ad463fc38941" Jul 28 01:18:27.843018 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.842950 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8bd7ba899dcd6c1360cd73b8222d7896742f7b24c9ed0cee2a4ad463fc38941"} err="failed to get container status \"b8bd7ba899dcd6c1360cd73b8222d7896742f7b24c9ed0cee2a4ad463fc38941\": rpc error: code = NotFound desc = could not find container \"b8bd7ba899dcd6c1360cd73b8222d7896742f7b24c9ed0cee2a4ad463fc38941\": container with ID starting with b8bd7ba899dcd6c1360cd73b8222d7896742f7b24c9ed0cee2a4ad463fc38941 not found: ID does not exist" Jul 28 01:18:27.843018 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.842977 2565 scope.go:117] "RemoveContainer" containerID="1f0b620a0b494ce7febf57d043be1519a66dcd33a0f16e0cc942804ac597065d" Jul 28 01:18:27.843248 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:18:27.843230 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f0b620a0b494ce7febf57d043be1519a66dcd33a0f16e0cc942804ac597065d\": container with ID starting with 1f0b620a0b494ce7febf57d043be1519a66dcd33a0f16e0cc942804ac597065d not found: ID does not exist" containerID="1f0b620a0b494ce7febf57d043be1519a66dcd33a0f16e0cc942804ac597065d" Jul 28 01:18:27.843283 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.843256 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f0b620a0b494ce7febf57d043be1519a66dcd33a0f16e0cc942804ac597065d"} err="failed to get container status \"1f0b620a0b494ce7febf57d043be1519a66dcd33a0f16e0cc942804ac597065d\": rpc error: code = NotFound desc = could not find container \"1f0b620a0b494ce7febf57d043be1519a66dcd33a0f16e0cc942804ac597065d\": container with ID starting with 1f0b620a0b494ce7febf57d043be1519a66dcd33a0f16e0cc942804ac597065d not found: ID does not exist" Jul 28 01:18:27.843283 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.843278 2565 scope.go:117] "RemoveContainer" containerID="c89a4b3983400d5340c36ea23dd217fb99d271e4eb9bbbf7e61ac902fa2f4426" Jul 28 01:18:27.843510 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:18:27.843490 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c89a4b3983400d5340c36ea23dd217fb99d271e4eb9bbbf7e61ac902fa2f4426\": container with ID starting with c89a4b3983400d5340c36ea23dd217fb99d271e4eb9bbbf7e61ac902fa2f4426 not found: ID does not exist" containerID="c89a4b3983400d5340c36ea23dd217fb99d271e4eb9bbbf7e61ac902fa2f4426" Jul 28 01:18:27.843553 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.843516 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c89a4b3983400d5340c36ea23dd217fb99d271e4eb9bbbf7e61ac902fa2f4426"} err="failed to get container status \"c89a4b3983400d5340c36ea23dd217fb99d271e4eb9bbbf7e61ac902fa2f4426\": rpc error: code = NotFound desc = could not find container \"c89a4b3983400d5340c36ea23dd217fb99d271e4eb9bbbf7e61ac902fa2f4426\": container with ID starting with c89a4b3983400d5340c36ea23dd217fb99d271e4eb9bbbf7e61ac902fa2f4426 not found: ID does not exist" Jul 28 01:18:27.843553 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.843532 2565 scope.go:117] "RemoveContainer" containerID="e04d3f6243d575ade0a0bad14f100b670a0f8cb7901980e2ccdc9da75851a978" Jul 28 01:18:27.843727 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:18:27.843712 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e04d3f6243d575ade0a0bad14f100b670a0f8cb7901980e2ccdc9da75851a978\": container with ID starting with e04d3f6243d575ade0a0bad14f100b670a0f8cb7901980e2ccdc9da75851a978 not found: ID does not exist" containerID="e04d3f6243d575ade0a0bad14f100b670a0f8cb7901980e2ccdc9da75851a978" Jul 28 01:18:27.843767 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.843730 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e04d3f6243d575ade0a0bad14f100b670a0f8cb7901980e2ccdc9da75851a978"} err="failed to get container status \"e04d3f6243d575ade0a0bad14f100b670a0f8cb7901980e2ccdc9da75851a978\": rpc error: code = NotFound desc = could not find container \"e04d3f6243d575ade0a0bad14f100b670a0f8cb7901980e2ccdc9da75851a978\": container with ID starting with e04d3f6243d575ade0a0bad14f100b670a0f8cb7901980e2ccdc9da75851a978 not found: ID does not exist" Jul 28 01:18:27.843767 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.843746 2565 scope.go:117] "RemoveContainer" containerID="8452ca82f62fc794efb7916e6fbdc4c8a40ce54e6c34c118adc1d26cb1af8d7a" Jul 28 01:18:27.843943 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:18:27.843926 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8452ca82f62fc794efb7916e6fbdc4c8a40ce54e6c34c118adc1d26cb1af8d7a\": container with ID starting with 8452ca82f62fc794efb7916e6fbdc4c8a40ce54e6c34c118adc1d26cb1af8d7a not found: ID does not exist" containerID="8452ca82f62fc794efb7916e6fbdc4c8a40ce54e6c34c118adc1d26cb1af8d7a" Jul 28 01:18:27.844003 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.843950 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8452ca82f62fc794efb7916e6fbdc4c8a40ce54e6c34c118adc1d26cb1af8d7a"} err="failed to get container status \"8452ca82f62fc794efb7916e6fbdc4c8a40ce54e6c34c118adc1d26cb1af8d7a\": rpc error: code = NotFound desc = could not find container \"8452ca82f62fc794efb7916e6fbdc4c8a40ce54e6c34c118adc1d26cb1af8d7a\": container with ID starting with 8452ca82f62fc794efb7916e6fbdc4c8a40ce54e6c34c118adc1d26cb1af8d7a not found: ID does not exist" Jul 28 01:18:27.844003 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.843972 2565 scope.go:117] "RemoveContainer" containerID="f897ff2800b97ed58dcd026e41e0d271020f0a3ff27120996cc7e8a440da1fba" Jul 28 01:18:27.844188 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:18:27.844173 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f897ff2800b97ed58dcd026e41e0d271020f0a3ff27120996cc7e8a440da1fba\": container with ID starting with f897ff2800b97ed58dcd026e41e0d271020f0a3ff27120996cc7e8a440da1fba not found: ID does not exist" containerID="f897ff2800b97ed58dcd026e41e0d271020f0a3ff27120996cc7e8a440da1fba" Jul 28 01:18:27.844224 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.844190 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f897ff2800b97ed58dcd026e41e0d271020f0a3ff27120996cc7e8a440da1fba"} err="failed to get container status \"f897ff2800b97ed58dcd026e41e0d271020f0a3ff27120996cc7e8a440da1fba\": rpc error: code = NotFound desc = could not find container \"f897ff2800b97ed58dcd026e41e0d271020f0a3ff27120996cc7e8a440da1fba\": container with ID starting with f897ff2800b97ed58dcd026e41e0d271020f0a3ff27120996cc7e8a440da1fba not found: ID does not exist" Jul 28 01:18:27.844224 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.844202 2565 scope.go:117] "RemoveContainer" containerID="a65698339fe1b14f410660bd154f0fcc1a62d76301ef37273b0e0f17ee853650" Jul 28 01:18:27.844394 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:18:27.844377 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a65698339fe1b14f410660bd154f0fcc1a62d76301ef37273b0e0f17ee853650\": container with ID starting with a65698339fe1b14f410660bd154f0fcc1a62d76301ef37273b0e0f17ee853650 not found: ID does not exist" containerID="a65698339fe1b14f410660bd154f0fcc1a62d76301ef37273b0e0f17ee853650" Jul 28 01:18:27.844438 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.844398 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a65698339fe1b14f410660bd154f0fcc1a62d76301ef37273b0e0f17ee853650"} err="failed to get container status \"a65698339fe1b14f410660bd154f0fcc1a62d76301ef37273b0e0f17ee853650\": rpc error: code = NotFound desc = could not find container \"a65698339fe1b14f410660bd154f0fcc1a62d76301ef37273b0e0f17ee853650\": container with ID starting with a65698339fe1b14f410660bd154f0fcc1a62d76301ef37273b0e0f17ee853650 not found: ID does not exist" Jul 28 01:18:27.852676 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.852657 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 01:18:27.852889 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.852878 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e290194-0951-4cc1-b02e-9f40fcf53cf0" containerName="kube-rbac-proxy-web" Jul 28 01:18:27.852928 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.852891 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e290194-0951-4cc1-b02e-9f40fcf53cf0" containerName="kube-rbac-proxy-web" Jul 28 01:18:27.852928 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.852898 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e290194-0951-4cc1-b02e-9f40fcf53cf0" containerName="kube-rbac-proxy-metric" Jul 28 01:18:27.852928 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.852904 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e290194-0951-4cc1-b02e-9f40fcf53cf0" containerName="kube-rbac-proxy-metric" Jul 28 01:18:27.852928 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.852912 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e290194-0951-4cc1-b02e-9f40fcf53cf0" containerName="config-reloader" Jul 28 01:18:27.852928 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.852918 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e290194-0951-4cc1-b02e-9f40fcf53cf0" containerName="config-reloader" Jul 28 01:18:27.852928 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.852929 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e290194-0951-4cc1-b02e-9f40fcf53cf0" containerName="init-config-reloader" Jul 28 01:18:27.853110 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.852935 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e290194-0951-4cc1-b02e-9f40fcf53cf0" containerName="init-config-reloader" Jul 28 01:18:27.853110 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.852941 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e290194-0951-4cc1-b02e-9f40fcf53cf0" containerName="prom-label-proxy" Jul 28 01:18:27.853110 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.852947 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e290194-0951-4cc1-b02e-9f40fcf53cf0" containerName="prom-label-proxy" Jul 28 01:18:27.853110 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.852955 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e290194-0951-4cc1-b02e-9f40fcf53cf0" containerName="alertmanager" Jul 28 01:18:27.853110 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.852961 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e290194-0951-4cc1-b02e-9f40fcf53cf0" containerName="alertmanager" Jul 28 01:18:27.853110 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.852966 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e290194-0951-4cc1-b02e-9f40fcf53cf0" containerName="kube-rbac-proxy" Jul 28 01:18:27.853110 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.852970 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e290194-0951-4cc1-b02e-9f40fcf53cf0" containerName="kube-rbac-proxy" Jul 28 01:18:27.853110 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.853009 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e290194-0951-4cc1-b02e-9f40fcf53cf0" containerName="config-reloader" Jul 28 01:18:27.853110 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.853016 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e290194-0951-4cc1-b02e-9f40fcf53cf0" containerName="kube-rbac-proxy" Jul 28 01:18:27.853110 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.853022 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e290194-0951-4cc1-b02e-9f40fcf53cf0" containerName="kube-rbac-proxy-web" Jul 28 01:18:27.853110 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.853045 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e290194-0951-4cc1-b02e-9f40fcf53cf0" containerName="alertmanager" Jul 28 01:18:27.853110 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.853051 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e290194-0951-4cc1-b02e-9f40fcf53cf0" containerName="prom-label-proxy" Jul 28 01:18:27.853110 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.853056 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e290194-0951-4cc1-b02e-9f40fcf53cf0" containerName="kube-rbac-proxy-metric" Jul 28 01:18:27.856146 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.856129 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:27.858789 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.858766 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 28 01:18:27.858896 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.858766 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 28 01:18:27.858896 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.858803 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-444q2\"" Jul 28 01:18:27.858896 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.858845 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 28 01:18:27.859259 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.859242 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 28 01:18:27.859303 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.859261 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 28 01:18:27.859386 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.859371 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 28 01:18:27.859502 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.859487 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 28 01:18:27.859552 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.859491 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 28 01:18:27.864391 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.864345 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 28 01:18:27.869744 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.869721 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 01:18:27.941523 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.941485 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/57766b8d-735c-444b-8fde-5ac7e36a037d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:27.941523 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.941529 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/57766b8d-735c-444b-8fde-5ac7e36a037d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:27.941743 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.941554 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r492\" (UniqueName: \"kubernetes.io/projected/57766b8d-735c-444b-8fde-5ac7e36a037d-kube-api-access-4r492\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:27.941743 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.941621 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/57766b8d-735c-444b-8fde-5ac7e36a037d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:27.941743 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.941655 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/57766b8d-735c-444b-8fde-5ac7e36a037d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:27.941743 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.941697 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/57766b8d-735c-444b-8fde-5ac7e36a037d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:27.941743 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.941721 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/57766b8d-735c-444b-8fde-5ac7e36a037d-web-config\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:27.941931 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.941745 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/57766b8d-735c-444b-8fde-5ac7e36a037d-config-volume\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:27.941931 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.941775 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57766b8d-735c-444b-8fde-5ac7e36a037d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:27.941931 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.941797 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57766b8d-735c-444b-8fde-5ac7e36a037d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:27.941931 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.941838 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/57766b8d-735c-444b-8fde-5ac7e36a037d-config-out\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:27.941931 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.941863 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/57766b8d-735c-444b-8fde-5ac7e36a037d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:27.941931 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:27.941895 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/57766b8d-735c-444b-8fde-5ac7e36a037d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:28.042363 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:28.042283 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/57766b8d-735c-444b-8fde-5ac7e36a037d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:28.042363 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:28.042323 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/57766b8d-735c-444b-8fde-5ac7e36a037d-web-config\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:28.042363 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:28.042340 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/57766b8d-735c-444b-8fde-5ac7e36a037d-config-volume\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:28.042639 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:28.042372 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57766b8d-735c-444b-8fde-5ac7e36a037d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:28.042639 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:28.042401 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57766b8d-735c-444b-8fde-5ac7e36a037d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:28.042639 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:28.042433 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/57766b8d-735c-444b-8fde-5ac7e36a037d-config-out\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:28.042639 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:28.042458 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/57766b8d-735c-444b-8fde-5ac7e36a037d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:28.042639 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:28.042493 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/57766b8d-735c-444b-8fde-5ac7e36a037d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:28.042639 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:28.042525 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/57766b8d-735c-444b-8fde-5ac7e36a037d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:28.042639 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:28.042549 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/57766b8d-735c-444b-8fde-5ac7e36a037d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:28.042639 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:28.042573 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r492\" (UniqueName: \"kubernetes.io/projected/57766b8d-735c-444b-8fde-5ac7e36a037d-kube-api-access-4r492\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:28.042639 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:28.042594 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/57766b8d-735c-444b-8fde-5ac7e36a037d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:28.042639 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:28.042622 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/57766b8d-735c-444b-8fde-5ac7e36a037d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:28.043569 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:28.043208 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57766b8d-735c-444b-8fde-5ac7e36a037d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:28.043569 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:28.043446 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57766b8d-735c-444b-8fde-5ac7e36a037d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:28.045388 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:28.045363 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/57766b8d-735c-444b-8fde-5ac7e36a037d-config-out\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:28.045491 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:28.045410 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/57766b8d-735c-444b-8fde-5ac7e36a037d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:28.045560 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:28.045518 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/57766b8d-735c-444b-8fde-5ac7e36a037d-web-config\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:28.045560 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:28.045518 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/57766b8d-735c-444b-8fde-5ac7e36a037d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:28.045664 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:28.045630 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/57766b8d-735c-444b-8fde-5ac7e36a037d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:28.045713 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:28.045629 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/57766b8d-735c-444b-8fde-5ac7e36a037d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:28.046250 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:28.046229 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/57766b8d-735c-444b-8fde-5ac7e36a037d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:28.046519 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:28.046498 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/57766b8d-735c-444b-8fde-5ac7e36a037d-config-volume\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:28.046564 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:28.046532 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/57766b8d-735c-444b-8fde-5ac7e36a037d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:28.047220 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:28.047198 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/57766b8d-735c-444b-8fde-5ac7e36a037d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:28.051412 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:28.051392 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r492\" (UniqueName: \"kubernetes.io/projected/57766b8d-735c-444b-8fde-5ac7e36a037d-kube-api-access-4r492\") pod \"alertmanager-main-0\" (UID: \"57766b8d-735c-444b-8fde-5ac7e36a037d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:28.165635 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:28.165592 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 01:18:28.305876 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:28.305796 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 01:18:28.308502 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:18:28.308466 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57766b8d_735c_444b_8fde_5ac7e36a037d.slice/crio-25bc6208fbcf9a4b1f2772dbcc04a4a7eab3d95c5779bbbaaba5a497b12c76ed WatchSource:0}: Error finding container 25bc6208fbcf9a4b1f2772dbcc04a4a7eab3d95c5779bbbaaba5a497b12c76ed: Status 404 returned error can't find the container with id 25bc6208fbcf9a4b1f2772dbcc04a4a7eab3d95c5779bbbaaba5a497b12c76ed Jul 28 01:18:28.801885 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:28.801851 2565 generic.go:356] "Generic (PLEG): container finished" podID="57766b8d-735c-444b-8fde-5ac7e36a037d" containerID="1a822d5508d3ebcb0a5b4fec0ebe4adc0ce45d75a0fd7a87ae155852216d8aa7" exitCode=0 Jul 28 01:18:28.802083 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:28.801918 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"57766b8d-735c-444b-8fde-5ac7e36a037d","Type":"ContainerDied","Data":"1a822d5508d3ebcb0a5b4fec0ebe4adc0ce45d75a0fd7a87ae155852216d8aa7"} Jul 28 01:18:28.802083 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:28.802015 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"57766b8d-735c-444b-8fde-5ac7e36a037d","Type":"ContainerStarted","Data":"25bc6208fbcf9a4b1f2772dbcc04a4a7eab3d95c5779bbbaaba5a497b12c76ed"} Jul 28 01:18:29.101644 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:29.101617 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e290194-0951-4cc1-b02e-9f40fcf53cf0" path="/var/lib/kubelet/pods/8e290194-0951-4cc1-b02e-9f40fcf53cf0/volumes" Jul 28 01:18:29.809440 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:29.809404 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"57766b8d-735c-444b-8fde-5ac7e36a037d","Type":"ContainerStarted","Data":"eb3d78f31f02a08843105ea8114aa8f6ee98079e00637f0fb9f23ad32fd43fdb"} Jul 28 01:18:29.809889 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:29.809466 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"57766b8d-735c-444b-8fde-5ac7e36a037d","Type":"ContainerStarted","Data":"6f7d534bcd997db1915d831da8d44403f9c692bd67e39dad8464f535d3d66789"} Jul 28 01:18:29.809889 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:29.809492 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"57766b8d-735c-444b-8fde-5ac7e36a037d","Type":"ContainerStarted","Data":"3e0d838e6c925d9cf45f8382a31243655a794d9949fa6926da79106cb5e5c217"} Jul 28 01:18:29.809889 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:29.809516 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"57766b8d-735c-444b-8fde-5ac7e36a037d","Type":"ContainerStarted","Data":"b45778682993b72ca19b5ecc3f5d3b728054e7445b164825b85ed134d52c6815"} Jul 28 01:18:29.809889 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:29.809539 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"57766b8d-735c-444b-8fde-5ac7e36a037d","Type":"ContainerStarted","Data":"07870463619a54c9aac62b1bc6adfa5266bff5e0105619b5bf03f189c93849d7"} Jul 28 01:18:29.809889 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:29.809569 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"57766b8d-735c-444b-8fde-5ac7e36a037d","Type":"ContainerStarted","Data":"97b09f8ab845d9366915ea665d229635850e9947e3d09e551acf4d9ae4325cc3"} Jul 28 01:18:29.833462 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:29.833404 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.818110751 podStartE2EDuration="2.83338734s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.276589ms" isStatefulPod=false podCreationTimestamp="2026-07-28 01:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:18:29.832744975 +0000 UTC m=+167.267143289" watchObservedRunningTime="2026-07-28 01:18:29.83338734 +0000 UTC m=+167.267785654" Jul 28 01:18:33.591078 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:33.591023 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58df9db6f4-kxmqm" Jul 28 01:18:33.591548 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:33.591108 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58df9db6f4-kxmqm" Jul 28 01:18:33.595693 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:33.595669 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58df9db6f4-kxmqm" Jul 28 01:18:33.828065 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:33.828026 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58df9db6f4-kxmqm" Jul 28 01:18:33.874167 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:33.874096 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f64cd8867-22rqd"] Jul 28 01:18:58.893979 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:58.893919 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5f64cd8867-22rqd" podUID="81615a60-9e59-4450-b53b-e36377150964" containerName="console" containerID="cri-o://3358945ec12917d200c27f0c5f7e8aa162edb822e892bd7b06ec3c539fd37872" gracePeriod=15 Jul 28 01:18:59.138688 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:59.138666 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f64cd8867-22rqd" Jul 28 01:18:59.269203 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:59.269173 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81615a60-9e59-4450-b53b-e36377150964-console-serving-cert\") pod \"81615a60-9e59-4450-b53b-e36377150964\" (UID: \"81615a60-9e59-4450-b53b-e36377150964\") " Jul 28 01:18:59.269346 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:59.269218 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81615a60-9e59-4450-b53b-e36377150964-oauth-serving-cert\") pod \"81615a60-9e59-4450-b53b-e36377150964\" (UID: \"81615a60-9e59-4450-b53b-e36377150964\") " Jul 28 01:18:59.269346 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:59.269256 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81615a60-9e59-4450-b53b-e36377150964-service-ca\") pod \"81615a60-9e59-4450-b53b-e36377150964\" (UID: \"81615a60-9e59-4450-b53b-e36377150964\") " Jul 28 01:18:59.269466 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:59.269419 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvh4z\" (UniqueName: \"kubernetes.io/projected/81615a60-9e59-4450-b53b-e36377150964-kube-api-access-rvh4z\") pod \"81615a60-9e59-4450-b53b-e36377150964\" (UID: \"81615a60-9e59-4450-b53b-e36377150964\") " Jul 28 01:18:59.269466 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:59.269459 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81615a60-9e59-4450-b53b-e36377150964-console-oauth-config\") pod \"81615a60-9e59-4450-b53b-e36377150964\" (UID: \"81615a60-9e59-4450-b53b-e36377150964\") " Jul 28 01:18:59.269576 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:59.269507 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81615a60-9e59-4450-b53b-e36377150964-trusted-ca-bundle\") pod \"81615a60-9e59-4450-b53b-e36377150964\" (UID: \"81615a60-9e59-4450-b53b-e36377150964\") " Jul 28 01:18:59.269576 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:59.269531 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81615a60-9e59-4450-b53b-e36377150964-console-config\") pod \"81615a60-9e59-4450-b53b-e36377150964\" (UID: \"81615a60-9e59-4450-b53b-e36377150964\") " Jul 28 01:18:59.269673 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:59.269634 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81615a60-9e59-4450-b53b-e36377150964-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "81615a60-9e59-4450-b53b-e36377150964" (UID: "81615a60-9e59-4450-b53b-e36377150964"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:18:59.269673 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:59.269643 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81615a60-9e59-4450-b53b-e36377150964-service-ca" (OuterVolumeSpecName: "service-ca") pod "81615a60-9e59-4450-b53b-e36377150964" (UID: "81615a60-9e59-4450-b53b-e36377150964"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:18:59.269920 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:59.269897 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81615a60-9e59-4450-b53b-e36377150964-service-ca\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:18:59.270107 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:59.269922 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81615a60-9e59-4450-b53b-e36377150964-oauth-serving-cert\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:18:59.270107 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:59.269920 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81615a60-9e59-4450-b53b-e36377150964-console-config" (OuterVolumeSpecName: "console-config") pod "81615a60-9e59-4450-b53b-e36377150964" (UID: "81615a60-9e59-4450-b53b-e36377150964"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:18:59.270107 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:59.269916 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81615a60-9e59-4450-b53b-e36377150964-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "81615a60-9e59-4450-b53b-e36377150964" (UID: "81615a60-9e59-4450-b53b-e36377150964"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:18:59.271331 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:59.271309 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81615a60-9e59-4450-b53b-e36377150964-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "81615a60-9e59-4450-b53b-e36377150964" (UID: "81615a60-9e59-4450-b53b-e36377150964"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:18:59.271450 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:59.271427 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81615a60-9e59-4450-b53b-e36377150964-kube-api-access-rvh4z" (OuterVolumeSpecName: "kube-api-access-rvh4z") pod "81615a60-9e59-4450-b53b-e36377150964" (UID: "81615a60-9e59-4450-b53b-e36377150964"). InnerVolumeSpecName "kube-api-access-rvh4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:18:59.271498 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:59.271461 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81615a60-9e59-4450-b53b-e36377150964-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "81615a60-9e59-4450-b53b-e36377150964" (UID: "81615a60-9e59-4450-b53b-e36377150964"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:18:59.370628 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:59.370594 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81615a60-9e59-4450-b53b-e36377150964-console-serving-cert\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:18:59.370628 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:59.370622 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvh4z\" (UniqueName: \"kubernetes.io/projected/81615a60-9e59-4450-b53b-e36377150964-kube-api-access-rvh4z\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:18:59.370628 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:59.370632 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81615a60-9e59-4450-b53b-e36377150964-console-oauth-config\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:18:59.370830 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:59.370641 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81615a60-9e59-4450-b53b-e36377150964-trusted-ca-bundle\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:18:59.370830 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:59.370650 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81615a60-9e59-4450-b53b-e36377150964-console-config\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:18:59.926048 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:59.925999 2565 generic.go:356] "Generic (PLEG): container finished" podID="81615a60-9e59-4450-b53b-e36377150964" containerID="3358945ec12917d200c27f0c5f7e8aa162edb822e892bd7b06ec3c539fd37872" exitCode=0 Jul 28 01:18:59.926470 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:59.926070 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f64cd8867-22rqd" Jul 28 01:18:59.926470 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:59.926083 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f64cd8867-22rqd" event={"ID":"81615a60-9e59-4450-b53b-e36377150964","Type":"ContainerDied","Data":"3358945ec12917d200c27f0c5f7e8aa162edb822e892bd7b06ec3c539fd37872"} Jul 28 01:18:59.926470 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:59.926154 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f64cd8867-22rqd" event={"ID":"81615a60-9e59-4450-b53b-e36377150964","Type":"ContainerDied","Data":"7f1ee53833c7449adadd487f04cb043b8db347ec623111225b0aa3f2ec6d7f01"} Jul 28 01:18:59.926470 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:59.926175 2565 scope.go:117] "RemoveContainer" containerID="3358945ec12917d200c27f0c5f7e8aa162edb822e892bd7b06ec3c539fd37872" Jul 28 01:18:59.934344 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:59.934330 2565 scope.go:117] "RemoveContainer" containerID="3358945ec12917d200c27f0c5f7e8aa162edb822e892bd7b06ec3c539fd37872" Jul 28 01:18:59.934603 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:18:59.934584 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3358945ec12917d200c27f0c5f7e8aa162edb822e892bd7b06ec3c539fd37872\": container with ID starting with 3358945ec12917d200c27f0c5f7e8aa162edb822e892bd7b06ec3c539fd37872 not found: ID does not exist" containerID="3358945ec12917d200c27f0c5f7e8aa162edb822e892bd7b06ec3c539fd37872" Jul 28 01:18:59.934665 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:59.934611 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3358945ec12917d200c27f0c5f7e8aa162edb822e892bd7b06ec3c539fd37872"} err="failed to get container status \"3358945ec12917d200c27f0c5f7e8aa162edb822e892bd7b06ec3c539fd37872\": rpc error: code = NotFound desc = could not find container \"3358945ec12917d200c27f0c5f7e8aa162edb822e892bd7b06ec3c539fd37872\": container with ID starting with 3358945ec12917d200c27f0c5f7e8aa162edb822e892bd7b06ec3c539fd37872 not found: ID does not exist" Jul 28 01:18:59.945908 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:59.945887 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f64cd8867-22rqd"] Jul 28 01:18:59.951424 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:18:59.951404 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f64cd8867-22rqd"] Jul 28 01:19:01.100119 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:01.100075 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81615a60-9e59-4450-b53b-e36377150964" path="/var/lib/kubelet/pods/81615a60-9e59-4450-b53b-e36377150964/volumes" Jul 28 01:19:28.520939 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:28.520905 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b4529"] Jul 28 01:19:28.521356 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:28.521196 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81615a60-9e59-4450-b53b-e36377150964" containerName="console" Jul 28 01:19:28.521356 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:28.521208 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="81615a60-9e59-4450-b53b-e36377150964" containerName="console" Jul 28 01:19:28.521356 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:28.521251 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="81615a60-9e59-4450-b53b-e36377150964" containerName="console" Jul 28 01:19:28.523958 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:28.523941 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b4529" Jul 28 01:19:28.526428 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:28.526404 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r9kmm\"" Jul 28 01:19:28.526559 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:28.526412 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 28 01:19:28.527517 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:28.527500 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 28 01:19:28.531740 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:28.531713 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b4529"] Jul 28 01:19:28.697207 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:28.697165 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3890185b-ce56-4650-8e1e-2843d90b9856-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b4529\" (UID: \"3890185b-ce56-4650-8e1e-2843d90b9856\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b4529" Jul 28 01:19:28.697207 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:28.697209 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3890185b-ce56-4650-8e1e-2843d90b9856-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b4529\" (UID: \"3890185b-ce56-4650-8e1e-2843d90b9856\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b4529" Jul 28 01:19:28.697449 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:28.697283 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdkfh\" (UniqueName: \"kubernetes.io/projected/3890185b-ce56-4650-8e1e-2843d90b9856-kube-api-access-hdkfh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b4529\" (UID: \"3890185b-ce56-4650-8e1e-2843d90b9856\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b4529" Jul 28 01:19:28.798556 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:28.798459 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdkfh\" (UniqueName: \"kubernetes.io/projected/3890185b-ce56-4650-8e1e-2843d90b9856-kube-api-access-hdkfh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b4529\" (UID: \"3890185b-ce56-4650-8e1e-2843d90b9856\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b4529" Jul 28 01:19:28.798556 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:28.798543 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3890185b-ce56-4650-8e1e-2843d90b9856-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b4529\" (UID: \"3890185b-ce56-4650-8e1e-2843d90b9856\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b4529" Jul 28 01:19:28.798778 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:28.798564 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3890185b-ce56-4650-8e1e-2843d90b9856-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b4529\" (UID: \"3890185b-ce56-4650-8e1e-2843d90b9856\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b4529" Jul 28 01:19:28.798897 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:28.798881 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3890185b-ce56-4650-8e1e-2843d90b9856-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b4529\" (UID: \"3890185b-ce56-4650-8e1e-2843d90b9856\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b4529" Jul 28 01:19:28.798973 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:28.798946 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3890185b-ce56-4650-8e1e-2843d90b9856-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b4529\" (UID: \"3890185b-ce56-4650-8e1e-2843d90b9856\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b4529" Jul 28 01:19:28.807854 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:28.807831 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdkfh\" (UniqueName: \"kubernetes.io/projected/3890185b-ce56-4650-8e1e-2843d90b9856-kube-api-access-hdkfh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b4529\" (UID: \"3890185b-ce56-4650-8e1e-2843d90b9856\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b4529" Jul 28 01:19:28.833556 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:28.833519 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b4529" Jul 28 01:19:28.960053 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:28.960015 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b4529"] Jul 28 01:19:28.962499 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:19:28.962472 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3890185b_ce56_4650_8e1e_2843d90b9856.slice/crio-c63d4025303d55d396f54863d2625bc5aeef0c1c8906dd70ce20df681781507a WatchSource:0}: Error finding container c63d4025303d55d396f54863d2625bc5aeef0c1c8906dd70ce20df681781507a: Status 404 returned error can't find the container with id c63d4025303d55d396f54863d2625bc5aeef0c1c8906dd70ce20df681781507a Jul 28 01:19:29.030629 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:29.030599 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b4529" event={"ID":"3890185b-ce56-4650-8e1e-2843d90b9856","Type":"ContainerStarted","Data":"c63d4025303d55d396f54863d2625bc5aeef0c1c8906dd70ce20df681781507a"} Jul 28 01:19:36.058400 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:36.058355 2565 generic.go:356] "Generic (PLEG): container finished" podID="3890185b-ce56-4650-8e1e-2843d90b9856" containerID="d51b12b0212eaef84d97ec6b0f1903e06b6c9b22ec441a0dcaf71d2cf584b351" exitCode=0 Jul 28 01:19:36.058814 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:36.058430 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b4529" event={"ID":"3890185b-ce56-4650-8e1e-2843d90b9856","Type":"ContainerDied","Data":"d51b12b0212eaef84d97ec6b0f1903e06b6c9b22ec441a0dcaf71d2cf584b351"} Jul 28 01:19:39.071831 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:39.071791 2565 generic.go:356] "Generic (PLEG): container finished" podID="3890185b-ce56-4650-8e1e-2843d90b9856" containerID="c902265596a57ea23bfaf9de599e4dafd3eedc9fe90a96ff441b20af831a510a" exitCode=0 Jul 28 01:19:39.072235 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:39.071866 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b4529" event={"ID":"3890185b-ce56-4650-8e1e-2843d90b9856","Type":"ContainerDied","Data":"c902265596a57ea23bfaf9de599e4dafd3eedc9fe90a96ff441b20af831a510a"} Jul 28 01:19:47.102810 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:47.102721 2565 generic.go:356] "Generic (PLEG): container finished" podID="3890185b-ce56-4650-8e1e-2843d90b9856" containerID="f059bbaff2031202f8342fe4737d9faed939ae203975aa176205d339b20c70c5" exitCode=0 Jul 28 01:19:47.102810 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:47.102779 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b4529" event={"ID":"3890185b-ce56-4650-8e1e-2843d90b9856","Type":"ContainerDied","Data":"f059bbaff2031202f8342fe4737d9faed939ae203975aa176205d339b20c70c5"} Jul 28 01:19:48.233543 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:48.233513 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b4529" Jul 28 01:19:48.359194 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:48.359117 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3890185b-ce56-4650-8e1e-2843d90b9856-util\") pod \"3890185b-ce56-4650-8e1e-2843d90b9856\" (UID: \"3890185b-ce56-4650-8e1e-2843d90b9856\") " Jul 28 01:19:48.359194 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:48.359156 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdkfh\" (UniqueName: \"kubernetes.io/projected/3890185b-ce56-4650-8e1e-2843d90b9856-kube-api-access-hdkfh\") pod \"3890185b-ce56-4650-8e1e-2843d90b9856\" (UID: \"3890185b-ce56-4650-8e1e-2843d90b9856\") " Jul 28 01:19:48.359194 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:48.359182 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3890185b-ce56-4650-8e1e-2843d90b9856-bundle\") pod \"3890185b-ce56-4650-8e1e-2843d90b9856\" (UID: \"3890185b-ce56-4650-8e1e-2843d90b9856\") " Jul 28 01:19:48.359849 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:48.359822 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3890185b-ce56-4650-8e1e-2843d90b9856-bundle" (OuterVolumeSpecName: "bundle") pod "3890185b-ce56-4650-8e1e-2843d90b9856" (UID: "3890185b-ce56-4650-8e1e-2843d90b9856"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:19:48.361378 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:48.361348 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3890185b-ce56-4650-8e1e-2843d90b9856-kube-api-access-hdkfh" (OuterVolumeSpecName: "kube-api-access-hdkfh") pod "3890185b-ce56-4650-8e1e-2843d90b9856" (UID: "3890185b-ce56-4650-8e1e-2843d90b9856"). InnerVolumeSpecName "kube-api-access-hdkfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:19:48.363706 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:48.363688 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3890185b-ce56-4650-8e1e-2843d90b9856-util" (OuterVolumeSpecName: "util") pod "3890185b-ce56-4650-8e1e-2843d90b9856" (UID: "3890185b-ce56-4650-8e1e-2843d90b9856"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:19:48.460127 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:48.460100 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3890185b-ce56-4650-8e1e-2843d90b9856-util\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:19:48.460127 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:48.460124 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdkfh\" (UniqueName: \"kubernetes.io/projected/3890185b-ce56-4650-8e1e-2843d90b9856-kube-api-access-hdkfh\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:19:48.460127 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:48.460133 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3890185b-ce56-4650-8e1e-2843d90b9856-bundle\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:19:49.110844 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:49.110815 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b4529" event={"ID":"3890185b-ce56-4650-8e1e-2843d90b9856","Type":"ContainerDied","Data":"c63d4025303d55d396f54863d2625bc5aeef0c1c8906dd70ce20df681781507a"} Jul 28 01:19:49.111015 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:49.110860 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c63d4025303d55d396f54863d2625bc5aeef0c1c8906dd70ce20df681781507a" Jul 28 01:19:49.111015 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:49.110828 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b4529" Jul 28 01:19:56.482653 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:56.482619 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-plcvd"] Jul 28 01:19:56.483026 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:56.482859 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3890185b-ce56-4650-8e1e-2843d90b9856" containerName="pull" Jul 28 01:19:56.483026 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:56.482868 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3890185b-ce56-4650-8e1e-2843d90b9856" containerName="pull" Jul 28 01:19:56.483026 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:56.482879 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3890185b-ce56-4650-8e1e-2843d90b9856" containerName="util" Jul 28 01:19:56.483026 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:56.482885 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3890185b-ce56-4650-8e1e-2843d90b9856" containerName="util" Jul 28 01:19:56.483026 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:56.482892 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3890185b-ce56-4650-8e1e-2843d90b9856" containerName="extract" Jul 28 01:19:56.483026 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:56.482897 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3890185b-ce56-4650-8e1e-2843d90b9856" containerName="extract" Jul 28 01:19:56.483026 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:56.482936 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3890185b-ce56-4650-8e1e-2843d90b9856" containerName="extract" Jul 28 01:19:56.486628 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:56.486611 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-plcvd" Jul 28 01:19:56.489178 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:56.489157 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 28 01:19:56.489759 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:56.489741 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-9qlnk\"" Jul 28 01:19:56.489835 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:56.489803 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 28 01:19:56.498870 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:56.498849 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-plcvd"] Jul 28 01:19:56.624399 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:56.624365 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgrkd\" (UniqueName: \"kubernetes.io/projected/571bbc3d-1515-45a9-bfc6-96fe367c0f49-kube-api-access-zgrkd\") pod \"cert-manager-operator-controller-manager-55c444bf86-plcvd\" (UID: \"571bbc3d-1515-45a9-bfc6-96fe367c0f49\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-plcvd" Jul 28 01:19:56.624556 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:56.624414 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/571bbc3d-1515-45a9-bfc6-96fe367c0f49-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-plcvd\" (UID: \"571bbc3d-1515-45a9-bfc6-96fe367c0f49\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-plcvd" Jul 28 01:19:56.725707 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:56.725671 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgrkd\" (UniqueName: \"kubernetes.io/projected/571bbc3d-1515-45a9-bfc6-96fe367c0f49-kube-api-access-zgrkd\") pod \"cert-manager-operator-controller-manager-55c444bf86-plcvd\" (UID: \"571bbc3d-1515-45a9-bfc6-96fe367c0f49\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-plcvd" Jul 28 01:19:56.725874 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:56.725721 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/571bbc3d-1515-45a9-bfc6-96fe367c0f49-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-plcvd\" (UID: \"571bbc3d-1515-45a9-bfc6-96fe367c0f49\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-plcvd" Jul 28 01:19:56.726077 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:56.726061 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/571bbc3d-1515-45a9-bfc6-96fe367c0f49-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-plcvd\" (UID: \"571bbc3d-1515-45a9-bfc6-96fe367c0f49\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-plcvd" Jul 28 01:19:56.734728 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:56.734674 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgrkd\" (UniqueName: \"kubernetes.io/projected/571bbc3d-1515-45a9-bfc6-96fe367c0f49-kube-api-access-zgrkd\") pod \"cert-manager-operator-controller-manager-55c444bf86-plcvd\" (UID: \"571bbc3d-1515-45a9-bfc6-96fe367c0f49\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-plcvd" Jul 28 01:19:56.795620 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:56.795591 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-plcvd" Jul 28 01:19:56.928561 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:56.928533 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-plcvd"] Jul 28 01:19:56.932325 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:19:56.932301 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod571bbc3d_1515_45a9_bfc6_96fe367c0f49.slice/crio-b84e319e293625c342e65e8ccc38b43de394aa41215a14f1068038448802eaae WatchSource:0}: Error finding container b84e319e293625c342e65e8ccc38b43de394aa41215a14f1068038448802eaae: Status 404 returned error can't find the container with id b84e319e293625c342e65e8ccc38b43de394aa41215a14f1068038448802eaae Jul 28 01:19:57.137084 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:57.136992 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-plcvd" event={"ID":"571bbc3d-1515-45a9-bfc6-96fe367c0f49","Type":"ContainerStarted","Data":"b84e319e293625c342e65e8ccc38b43de394aa41215a14f1068038448802eaae"} Jul 28 01:19:59.145726 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:59.145693 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-plcvd" event={"ID":"571bbc3d-1515-45a9-bfc6-96fe367c0f49","Type":"ContainerStarted","Data":"2680bdf44f7aaa7f44b0fd6ca380ec69e5f4c49129324b5a53c9fb89ef6e7f8f"} Jul 28 01:19:59.174136 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:19:59.174075 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-plcvd" podStartSLOduration=1.452664741 podStartE2EDuration="3.174056535s" totalImagesPullingTime="1.721391794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:19:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:19:59.171547169 +0000 UTC m=+256.605945484" watchObservedRunningTime="2026-07-28 01:19:59.174056535 +0000 UTC m=+256.608454847" Jul 28 01:20:00.167879 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:00.167846 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbv9v7"] Jul 28 01:20:00.170076 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:00.170059 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbv9v7" Jul 28 01:20:00.172777 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:00.172758 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 28 01:20:00.172918 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:00.172761 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r9kmm\"" Jul 28 01:20:00.172918 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:00.172790 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 28 01:20:00.179737 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:00.179712 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbv9v7"] Jul 28 01:20:00.253657 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:00.253614 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/810e58ca-1925-46be-bec0-6fea07c55dc3-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbv9v7\" (UID: \"810e58ca-1925-46be-bec0-6fea07c55dc3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbv9v7" Jul 28 01:20:00.253820 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:00.253673 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh5nr\" (UniqueName: \"kubernetes.io/projected/810e58ca-1925-46be-bec0-6fea07c55dc3-kube-api-access-lh5nr\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbv9v7\" (UID: \"810e58ca-1925-46be-bec0-6fea07c55dc3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbv9v7" Jul 28 01:20:00.253820 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:00.253769 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/810e58ca-1925-46be-bec0-6fea07c55dc3-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbv9v7\" (UID: \"810e58ca-1925-46be-bec0-6fea07c55dc3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbv9v7" Jul 28 01:20:00.354994 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:00.354957 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/810e58ca-1925-46be-bec0-6fea07c55dc3-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbv9v7\" (UID: \"810e58ca-1925-46be-bec0-6fea07c55dc3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbv9v7" Jul 28 01:20:00.355198 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:00.355025 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/810e58ca-1925-46be-bec0-6fea07c55dc3-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbv9v7\" (UID: \"810e58ca-1925-46be-bec0-6fea07c55dc3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbv9v7" Jul 28 01:20:00.355198 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:00.355068 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh5nr\" (UniqueName: \"kubernetes.io/projected/810e58ca-1925-46be-bec0-6fea07c55dc3-kube-api-access-lh5nr\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbv9v7\" (UID: \"810e58ca-1925-46be-bec0-6fea07c55dc3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbv9v7" Jul 28 01:20:00.355401 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:00.355380 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/810e58ca-1925-46be-bec0-6fea07c55dc3-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbv9v7\" (UID: \"810e58ca-1925-46be-bec0-6fea07c55dc3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbv9v7" Jul 28 01:20:00.355469 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:00.355453 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/810e58ca-1925-46be-bec0-6fea07c55dc3-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbv9v7\" (UID: \"810e58ca-1925-46be-bec0-6fea07c55dc3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbv9v7" Jul 28 01:20:00.368894 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:00.368869 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh5nr\" (UniqueName: \"kubernetes.io/projected/810e58ca-1925-46be-bec0-6fea07c55dc3-kube-api-access-lh5nr\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbv9v7\" (UID: \"810e58ca-1925-46be-bec0-6fea07c55dc3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbv9v7" Jul 28 01:20:00.479287 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:00.479255 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbv9v7" Jul 28 01:20:00.604678 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:00.604641 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbv9v7"] Jul 28 01:20:00.607454 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:20:00.607425 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod810e58ca_1925_46be_bec0_6fea07c55dc3.slice/crio-bf8d0189302bca0ad96c5cf8fba49bde60ac10a9fc1ceb781722175b82846d40 WatchSource:0}: Error finding container bf8d0189302bca0ad96c5cf8fba49bde60ac10a9fc1ceb781722175b82846d40: Status 404 returned error can't find the container with id bf8d0189302bca0ad96c5cf8fba49bde60ac10a9fc1ceb781722175b82846d40 Jul 28 01:20:01.153981 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:01.153948 2565 generic.go:356] "Generic (PLEG): container finished" podID="810e58ca-1925-46be-bec0-6fea07c55dc3" containerID="ee9feaf287ad5de4fc8cb58ba879c477b34cb05b855d05a23b7a2704d1a97637" exitCode=0 Jul 28 01:20:01.154171 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:01.154049 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbv9v7" event={"ID":"810e58ca-1925-46be-bec0-6fea07c55dc3","Type":"ContainerDied","Data":"ee9feaf287ad5de4fc8cb58ba879c477b34cb05b855d05a23b7a2704d1a97637"} Jul 28 01:20:01.154171 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:01.154095 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbv9v7" event={"ID":"810e58ca-1925-46be-bec0-6fea07c55dc3","Type":"ContainerStarted","Data":"bf8d0189302bca0ad96c5cf8fba49bde60ac10a9fc1ceb781722175b82846d40"} Jul 28 01:20:01.319947 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:01.319912 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-ts6wn"] Jul 28 01:20:01.321866 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:01.321846 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-ts6wn" Jul 28 01:20:01.324362 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:01.324341 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 28 01:20:01.324476 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:01.324349 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 28 01:20:01.324530 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:01.324472 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-c8928\"" Jul 28 01:20:01.331004 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:01.330985 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-ts6wn"] Jul 28 01:20:01.463910 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:01.463867 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qhjw\" (UniqueName: \"kubernetes.io/projected/09e071d6-7aa4-4775-940c-eb775172f9ed-kube-api-access-6qhjw\") pod \"cert-manager-webhook-6486ffd997-ts6wn\" (UID: \"09e071d6-7aa4-4775-940c-eb775172f9ed\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ts6wn" Jul 28 01:20:01.464107 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:01.463949 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09e071d6-7aa4-4775-940c-eb775172f9ed-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-ts6wn\" (UID: \"09e071d6-7aa4-4775-940c-eb775172f9ed\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ts6wn" Jul 28 01:20:01.564927 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:01.564890 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09e071d6-7aa4-4775-940c-eb775172f9ed-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-ts6wn\" (UID: \"09e071d6-7aa4-4775-940c-eb775172f9ed\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ts6wn" Jul 28 01:20:01.565129 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:01.564954 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qhjw\" (UniqueName: \"kubernetes.io/projected/09e071d6-7aa4-4775-940c-eb775172f9ed-kube-api-access-6qhjw\") pod \"cert-manager-webhook-6486ffd997-ts6wn\" (UID: \"09e071d6-7aa4-4775-940c-eb775172f9ed\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ts6wn" Jul 28 01:20:01.573591 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:01.573561 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09e071d6-7aa4-4775-940c-eb775172f9ed-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-ts6wn\" (UID: \"09e071d6-7aa4-4775-940c-eb775172f9ed\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ts6wn" Jul 28 01:20:01.573845 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:01.573822 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qhjw\" (UniqueName: \"kubernetes.io/projected/09e071d6-7aa4-4775-940c-eb775172f9ed-kube-api-access-6qhjw\") pod \"cert-manager-webhook-6486ffd997-ts6wn\" (UID: \"09e071d6-7aa4-4775-940c-eb775172f9ed\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ts6wn" Jul 28 01:20:01.643282 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:01.643244 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-ts6wn" Jul 28 01:20:01.767761 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:01.767739 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-ts6wn"] Jul 28 01:20:01.770174 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:20:01.770147 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09e071d6_7aa4_4775_940c_eb775172f9ed.slice/crio-453447904d3aa335e8dd7780afe0fe1edbf17ef249bbe33aa659d4a590a86f17 WatchSource:0}: Error finding container 453447904d3aa335e8dd7780afe0fe1edbf17ef249bbe33aa659d4a590a86f17: Status 404 returned error can't find the container with id 453447904d3aa335e8dd7780afe0fe1edbf17ef249bbe33aa659d4a590a86f17 Jul 28 01:20:02.158738 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:02.158650 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-ts6wn" event={"ID":"09e071d6-7aa4-4775-940c-eb775172f9ed","Type":"ContainerStarted","Data":"453447904d3aa335e8dd7780afe0fe1edbf17ef249bbe33aa659d4a590a86f17"} Jul 28 01:20:02.626192 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:02.624354 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-xrztm"] Jul 28 01:20:02.628201 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:02.628173 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xrztm" Jul 28 01:20:02.630690 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:02.630654 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-j96z2\"" Jul 28 01:20:02.633556 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:02.633530 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-xrztm"] Jul 28 01:20:02.776971 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:02.776936 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a532cc31-a8be-4f23-82a6-40b68449ec34-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-xrztm\" (UID: \"a532cc31-a8be-4f23-82a6-40b68449ec34\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xrztm" Jul 28 01:20:02.777130 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:02.776977 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9stf8\" (UniqueName: \"kubernetes.io/projected/a532cc31-a8be-4f23-82a6-40b68449ec34-kube-api-access-9stf8\") pod \"cert-manager-cainjector-559cdd8c87-xrztm\" (UID: \"a532cc31-a8be-4f23-82a6-40b68449ec34\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xrztm" Jul 28 01:20:02.877647 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:02.877555 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9stf8\" (UniqueName: \"kubernetes.io/projected/a532cc31-a8be-4f23-82a6-40b68449ec34-kube-api-access-9stf8\") pod \"cert-manager-cainjector-559cdd8c87-xrztm\" (UID: \"a532cc31-a8be-4f23-82a6-40b68449ec34\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xrztm" Jul 28 01:20:02.877647 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:02.877635 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a532cc31-a8be-4f23-82a6-40b68449ec34-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-xrztm\" (UID: \"a532cc31-a8be-4f23-82a6-40b68449ec34\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xrztm" Jul 28 01:20:02.887324 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:02.887289 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a532cc31-a8be-4f23-82a6-40b68449ec34-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-xrztm\" (UID: \"a532cc31-a8be-4f23-82a6-40b68449ec34\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xrztm" Jul 28 01:20:02.887474 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:02.887305 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9stf8\" (UniqueName: \"kubernetes.io/projected/a532cc31-a8be-4f23-82a6-40b68449ec34-kube-api-access-9stf8\") pod \"cert-manager-cainjector-559cdd8c87-xrztm\" (UID: \"a532cc31-a8be-4f23-82a6-40b68449ec34\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xrztm" Jul 28 01:20:02.938092 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:02.938057 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xrztm" Jul 28 01:20:03.086131 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:03.086094 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-xrztm"] Jul 28 01:20:03.088952 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:20:03.088922 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda532cc31_a8be_4f23_82a6_40b68449ec34.slice/crio-3361239a1ae1cddb1430d6f5a72229310ca699f260a8934d6b83a99009f278ae WatchSource:0}: Error finding container 3361239a1ae1cddb1430d6f5a72229310ca699f260a8934d6b83a99009f278ae: Status 404 returned error can't find the container with id 3361239a1ae1cddb1430d6f5a72229310ca699f260a8934d6b83a99009f278ae Jul 28 01:20:03.165215 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:03.165130 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xrztm" event={"ID":"a532cc31-a8be-4f23-82a6-40b68449ec34","Type":"ContainerStarted","Data":"3361239a1ae1cddb1430d6f5a72229310ca699f260a8934d6b83a99009f278ae"} Jul 28 01:20:05.175806 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:05.175650 2565 generic.go:356] "Generic (PLEG): container finished" podID="810e58ca-1925-46be-bec0-6fea07c55dc3" containerID="7d6033f29e7ced921dd36cb57a28e8860668e10a690d693d70ec58aded4d94e6" exitCode=0 Jul 28 01:20:05.175806 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:05.175689 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbv9v7" event={"ID":"810e58ca-1925-46be-bec0-6fea07c55dc3","Type":"ContainerDied","Data":"7d6033f29e7ced921dd36cb57a28e8860668e10a690d693d70ec58aded4d94e6"} Jul 28 01:20:06.183308 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:06.183270 2565 generic.go:356] "Generic (PLEG): container finished" podID="810e58ca-1925-46be-bec0-6fea07c55dc3" containerID="c3c8285e17ed2479b98239a2549ae8de0229dfe165bfa25eec368f4dc417366a" exitCode=0 Jul 28 01:20:06.183721 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:06.183360 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbv9v7" event={"ID":"810e58ca-1925-46be-bec0-6fea07c55dc3","Type":"ContainerDied","Data":"c3c8285e17ed2479b98239a2549ae8de0229dfe165bfa25eec368f4dc417366a"} Jul 28 01:20:07.188443 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:07.188403 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-ts6wn" event={"ID":"09e071d6-7aa4-4775-940c-eb775172f9ed","Type":"ContainerStarted","Data":"0d89c9b0eee0a3db0781e076f58ea41aa9dcccd0e3ccc8d7d932a9853c6e111d"} Jul 28 01:20:07.188964 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:07.188496 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-ts6wn" Jul 28 01:20:07.190057 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:07.190010 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xrztm" event={"ID":"a532cc31-a8be-4f23-82a6-40b68449ec34","Type":"ContainerStarted","Data":"a2ee96d28d7e2171bfa784457db9be01e2fe6e722eb3bee92517e44a0f4f427a"} Jul 28 01:20:07.224288 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:07.224234 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-ts6wn" podStartSLOduration=1.305518886 podStartE2EDuration="6.224219129s" totalImagesPullingTime="4.918700243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:20:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:20:07.223088808 +0000 UTC m=+264.657487123" watchObservedRunningTime="2026-07-28 01:20:07.224219129 +0000 UTC m=+264.658617443" Jul 28 01:20:07.259269 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:07.259212 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xrztm" podStartSLOduration=1.666116829 podStartE2EDuration="5.259195805s" totalImagesPullingTime="3.593078976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:20:07.257247422 +0000 UTC m=+264.691645737" watchObservedRunningTime="2026-07-28 01:20:07.259195805 +0000 UTC m=+264.693594118" Jul 28 01:20:07.320908 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:07.320884 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbv9v7" Jul 28 01:20:07.417598 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:07.417566 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/810e58ca-1925-46be-bec0-6fea07c55dc3-util\") pod \"810e58ca-1925-46be-bec0-6fea07c55dc3\" (UID: \"810e58ca-1925-46be-bec0-6fea07c55dc3\") " Jul 28 01:20:07.417762 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:07.417679 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lh5nr\" (UniqueName: \"kubernetes.io/projected/810e58ca-1925-46be-bec0-6fea07c55dc3-kube-api-access-lh5nr\") pod \"810e58ca-1925-46be-bec0-6fea07c55dc3\" (UID: \"810e58ca-1925-46be-bec0-6fea07c55dc3\") " Jul 28 01:20:07.417762 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:07.417713 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/810e58ca-1925-46be-bec0-6fea07c55dc3-bundle\") pod \"810e58ca-1925-46be-bec0-6fea07c55dc3\" (UID: \"810e58ca-1925-46be-bec0-6fea07c55dc3\") " Jul 28 01:20:07.418124 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:07.418093 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/810e58ca-1925-46be-bec0-6fea07c55dc3-bundle" (OuterVolumeSpecName: "bundle") pod "810e58ca-1925-46be-bec0-6fea07c55dc3" (UID: "810e58ca-1925-46be-bec0-6fea07c55dc3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:20:07.419730 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:07.419704 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/810e58ca-1925-46be-bec0-6fea07c55dc3-kube-api-access-lh5nr" (OuterVolumeSpecName: "kube-api-access-lh5nr") pod "810e58ca-1925-46be-bec0-6fea07c55dc3" (UID: "810e58ca-1925-46be-bec0-6fea07c55dc3"). InnerVolumeSpecName "kube-api-access-lh5nr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:20:07.422184 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:07.422166 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/810e58ca-1925-46be-bec0-6fea07c55dc3-util" (OuterVolumeSpecName: "util") pod "810e58ca-1925-46be-bec0-6fea07c55dc3" (UID: "810e58ca-1925-46be-bec0-6fea07c55dc3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:20:07.519336 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:07.519249 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/810e58ca-1925-46be-bec0-6fea07c55dc3-util\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:20:07.519336 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:07.519277 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh5nr\" (UniqueName: \"kubernetes.io/projected/810e58ca-1925-46be-bec0-6fea07c55dc3-kube-api-access-lh5nr\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:20:07.519336 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:07.519288 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/810e58ca-1925-46be-bec0-6fea07c55dc3-bundle\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:20:08.195123 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:08.195093 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbv9v7" Jul 28 01:20:08.195507 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:08.195128 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbv9v7" event={"ID":"810e58ca-1925-46be-bec0-6fea07c55dc3","Type":"ContainerDied","Data":"bf8d0189302bca0ad96c5cf8fba49bde60ac10a9fc1ceb781722175b82846d40"} Jul 28 01:20:08.195507 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:08.195178 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf8d0189302bca0ad96c5cf8fba49bde60ac10a9fc1ceb781722175b82846d40" Jul 28 01:20:13.197743 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:13.197713 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-ts6wn" Jul 28 01:20:20.282009 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:20.281974 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj8pv6"] Jul 28 01:20:20.282409 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:20.282274 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="810e58ca-1925-46be-bec0-6fea07c55dc3" containerName="pull" Jul 28 01:20:20.282409 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:20.282285 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="810e58ca-1925-46be-bec0-6fea07c55dc3" containerName="pull" Jul 28 01:20:20.282409 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:20.282292 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="810e58ca-1925-46be-bec0-6fea07c55dc3" containerName="extract" Jul 28 01:20:20.282409 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:20.282297 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="810e58ca-1925-46be-bec0-6fea07c55dc3" containerName="extract" Jul 28 01:20:20.282409 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:20.282304 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="810e58ca-1925-46be-bec0-6fea07c55dc3" containerName="util" Jul 28 01:20:20.282409 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:20.282311 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="810e58ca-1925-46be-bec0-6fea07c55dc3" containerName="util" Jul 28 01:20:20.282409 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:20.282356 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="810e58ca-1925-46be-bec0-6fea07c55dc3" containerName="extract" Jul 28 01:20:20.286114 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:20.286096 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj8pv6" Jul 28 01:20:20.288662 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:20.288636 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 28 01:20:20.288805 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:20.288644 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 28 01:20:20.289695 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:20.289679 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r9kmm\"" Jul 28 01:20:20.292265 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:20.292243 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj8pv6"] Jul 28 01:20:20.422388 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:20.422342 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cwmr\" (UniqueName: \"kubernetes.io/projected/8b1d592a-aaac-4009-bdb9-2678e890ec70-kube-api-access-7cwmr\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj8pv6\" (UID: \"8b1d592a-aaac-4009-bdb9-2678e890ec70\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj8pv6" Jul 28 01:20:20.422388 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:20.422400 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b1d592a-aaac-4009-bdb9-2678e890ec70-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj8pv6\" (UID: \"8b1d592a-aaac-4009-bdb9-2678e890ec70\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj8pv6" Jul 28 01:20:20.422612 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:20.422491 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b1d592a-aaac-4009-bdb9-2678e890ec70-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj8pv6\" (UID: \"8b1d592a-aaac-4009-bdb9-2678e890ec70\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj8pv6" Jul 28 01:20:20.523674 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:20.523638 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b1d592a-aaac-4009-bdb9-2678e890ec70-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj8pv6\" (UID: \"8b1d592a-aaac-4009-bdb9-2678e890ec70\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj8pv6" Jul 28 01:20:20.523674 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:20.523677 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cwmr\" (UniqueName: \"kubernetes.io/projected/8b1d592a-aaac-4009-bdb9-2678e890ec70-kube-api-access-7cwmr\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj8pv6\" (UID: \"8b1d592a-aaac-4009-bdb9-2678e890ec70\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj8pv6" Jul 28 01:20:20.523895 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:20.523707 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b1d592a-aaac-4009-bdb9-2678e890ec70-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj8pv6\" (UID: \"8b1d592a-aaac-4009-bdb9-2678e890ec70\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj8pv6" Jul 28 01:20:20.524023 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:20.524002 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b1d592a-aaac-4009-bdb9-2678e890ec70-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj8pv6\" (UID: \"8b1d592a-aaac-4009-bdb9-2678e890ec70\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj8pv6" Jul 28 01:20:20.524165 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:20.524109 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b1d592a-aaac-4009-bdb9-2678e890ec70-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj8pv6\" (UID: \"8b1d592a-aaac-4009-bdb9-2678e890ec70\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj8pv6" Jul 28 01:20:20.534879 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:20.534821 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cwmr\" (UniqueName: \"kubernetes.io/projected/8b1d592a-aaac-4009-bdb9-2678e890ec70-kube-api-access-7cwmr\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj8pv6\" (UID: \"8b1d592a-aaac-4009-bdb9-2678e890ec70\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj8pv6" Jul 28 01:20:20.595407 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:20.595371 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj8pv6" Jul 28 01:20:20.719744 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:20.719710 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj8pv6"] Jul 28 01:20:20.722793 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:20:20.722762 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b1d592a_aaac_4009_bdb9_2678e890ec70.slice/crio-192bc21a8d3f1a3b65ef4d4cbd1ccca31f668591ff1e4b89edbcd176978ca6d2 WatchSource:0}: Error finding container 192bc21a8d3f1a3b65ef4d4cbd1ccca31f668591ff1e4b89edbcd176978ca6d2: Status 404 returned error can't find the container with id 192bc21a8d3f1a3b65ef4d4cbd1ccca31f668591ff1e4b89edbcd176978ca6d2 Jul 28 01:20:21.247658 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:21.247624 2565 generic.go:356] "Generic (PLEG): container finished" podID="8b1d592a-aaac-4009-bdb9-2678e890ec70" containerID="a09f4a11be9216a5bf25c97ac22d3a1e40a160a9601efeaca6909d3adbcbfae6" exitCode=0 Jul 28 01:20:21.247828 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:21.247710 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj8pv6" event={"ID":"8b1d592a-aaac-4009-bdb9-2678e890ec70","Type":"ContainerDied","Data":"a09f4a11be9216a5bf25c97ac22d3a1e40a160a9601efeaca6909d3adbcbfae6"} Jul 28 01:20:21.247828 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:21.247754 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj8pv6" event={"ID":"8b1d592a-aaac-4009-bdb9-2678e890ec70","Type":"ContainerStarted","Data":"192bc21a8d3f1a3b65ef4d4cbd1ccca31f668591ff1e4b89edbcd176978ca6d2"} Jul 28 01:20:22.252938 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:22.252903 2565 generic.go:356] "Generic (PLEG): container finished" podID="8b1d592a-aaac-4009-bdb9-2678e890ec70" containerID="d763b89559ce9720c5eb2c00406d642a62c04ae5c9c98c25dadb9379e8331520" exitCode=0 Jul 28 01:20:22.253328 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:22.252979 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj8pv6" event={"ID":"8b1d592a-aaac-4009-bdb9-2678e890ec70","Type":"ContainerDied","Data":"d763b89559ce9720c5eb2c00406d642a62c04ae5c9c98c25dadb9379e8331520"} Jul 28 01:20:23.259151 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:23.259115 2565 generic.go:356] "Generic (PLEG): container finished" podID="8b1d592a-aaac-4009-bdb9-2678e890ec70" containerID="8549d85b5aaf372e6d3e76d8c0e792554548d4bf61c2dcbb5ca54fa0520ac774" exitCode=0 Jul 28 01:20:23.259567 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:23.259172 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj8pv6" event={"ID":"8b1d592a-aaac-4009-bdb9-2678e890ec70","Type":"ContainerDied","Data":"8549d85b5aaf372e6d3e76d8c0e792554548d4bf61c2dcbb5ca54fa0520ac774"} Jul 28 01:20:24.388051 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:24.388005 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj8pv6" Jul 28 01:20:24.559094 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:24.558981 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b1d592a-aaac-4009-bdb9-2678e890ec70-bundle\") pod \"8b1d592a-aaac-4009-bdb9-2678e890ec70\" (UID: \"8b1d592a-aaac-4009-bdb9-2678e890ec70\") " Jul 28 01:20:24.559094 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:24.559056 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7cwmr\" (UniqueName: \"kubernetes.io/projected/8b1d592a-aaac-4009-bdb9-2678e890ec70-kube-api-access-7cwmr\") pod \"8b1d592a-aaac-4009-bdb9-2678e890ec70\" (UID: \"8b1d592a-aaac-4009-bdb9-2678e890ec70\") " Jul 28 01:20:24.559275 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:24.559106 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b1d592a-aaac-4009-bdb9-2678e890ec70-util\") pod \"8b1d592a-aaac-4009-bdb9-2678e890ec70\" (UID: \"8b1d592a-aaac-4009-bdb9-2678e890ec70\") " Jul 28 01:20:24.559744 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:24.559720 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1d592a-aaac-4009-bdb9-2678e890ec70-bundle" (OuterVolumeSpecName: "bundle") pod "8b1d592a-aaac-4009-bdb9-2678e890ec70" (UID: "8b1d592a-aaac-4009-bdb9-2678e890ec70"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:20:24.561163 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:24.561133 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b1d592a-aaac-4009-bdb9-2678e890ec70-kube-api-access-7cwmr" (OuterVolumeSpecName: "kube-api-access-7cwmr") pod "8b1d592a-aaac-4009-bdb9-2678e890ec70" (UID: "8b1d592a-aaac-4009-bdb9-2678e890ec70"). InnerVolumeSpecName "kube-api-access-7cwmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:20:24.565201 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:24.565176 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1d592a-aaac-4009-bdb9-2678e890ec70-util" (OuterVolumeSpecName: "util") pod "8b1d592a-aaac-4009-bdb9-2678e890ec70" (UID: "8b1d592a-aaac-4009-bdb9-2678e890ec70"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:20:24.660260 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:24.660225 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b1d592a-aaac-4009-bdb9-2678e890ec70-bundle\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:20:24.660260 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:24.660255 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cwmr\" (UniqueName: \"kubernetes.io/projected/8b1d592a-aaac-4009-bdb9-2678e890ec70-kube-api-access-7cwmr\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:20:24.660260 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:24.660265 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b1d592a-aaac-4009-bdb9-2678e890ec70-util\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:20:25.274709 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:25.274677 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj8pv6" event={"ID":"8b1d592a-aaac-4009-bdb9-2678e890ec70","Type":"ContainerDied","Data":"192bc21a8d3f1a3b65ef4d4cbd1ccca31f668591ff1e4b89edbcd176978ca6d2"} Jul 28 01:20:25.274872 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:25.274717 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afj8pv6" Jul 28 01:20:25.274872 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:25.274723 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="192bc21a8d3f1a3b65ef4d4cbd1ccca31f668591ff1e4b89edbcd176978ca6d2" Jul 28 01:20:37.403373 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:37.403342 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8699bb9d7c-6p952"] Jul 28 01:20:37.403753 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:37.403595 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b1d592a-aaac-4009-bdb9-2678e890ec70" containerName="util" Jul 28 01:20:37.403753 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:37.403606 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b1d592a-aaac-4009-bdb9-2678e890ec70" containerName="util" Jul 28 01:20:37.403753 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:37.403622 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b1d592a-aaac-4009-bdb9-2678e890ec70" containerName="pull" Jul 28 01:20:37.403753 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:37.403627 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b1d592a-aaac-4009-bdb9-2678e890ec70" containerName="pull" Jul 28 01:20:37.403753 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:37.403635 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b1d592a-aaac-4009-bdb9-2678e890ec70" containerName="extract" Jul 28 01:20:37.403753 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:37.403640 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b1d592a-aaac-4009-bdb9-2678e890ec70" containerName="extract" Jul 28 01:20:37.403753 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:37.403686 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b1d592a-aaac-4009-bdb9-2678e890ec70" containerName="extract" Jul 28 01:20:37.405812 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:37.405792 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8699bb9d7c-6p952" Jul 28 01:20:37.408373 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:37.408352 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 28 01:20:37.408523 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:37.408480 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-ndl4c\"" Jul 28 01:20:37.408523 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:37.408495 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 28 01:20:37.408653 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:37.408562 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 28 01:20:37.408653 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:37.408639 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 28 01:20:37.431012 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:37.430990 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8699bb9d7c-6p952"] Jul 28 01:20:37.463560 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:37.463532 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7fwz\" (UniqueName: \"kubernetes.io/projected/867b0a1e-9249-4fbc-be63-e0495f3aff8b-kube-api-access-p7fwz\") pod \"opendatahub-operator-controller-manager-8699bb9d7c-6p952\" (UID: \"867b0a1e-9249-4fbc-be63-e0495f3aff8b\") " pod="opendatahub/opendatahub-operator-controller-manager-8699bb9d7c-6p952" Jul 28 01:20:37.463665 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:37.463569 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/867b0a1e-9249-4fbc-be63-e0495f3aff8b-tmp\") pod \"opendatahub-operator-controller-manager-8699bb9d7c-6p952\" (UID: \"867b0a1e-9249-4fbc-be63-e0495f3aff8b\") " pod="opendatahub/opendatahub-operator-controller-manager-8699bb9d7c-6p952" Jul 28 01:20:37.463665 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:37.463591 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/867b0a1e-9249-4fbc-be63-e0495f3aff8b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8699bb9d7c-6p952\" (UID: \"867b0a1e-9249-4fbc-be63-e0495f3aff8b\") " pod="opendatahub/opendatahub-operator-controller-manager-8699bb9d7c-6p952" Jul 28 01:20:37.463737 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:37.463663 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/867b0a1e-9249-4fbc-be63-e0495f3aff8b-webhook-cert\") pod \"opendatahub-operator-controller-manager-8699bb9d7c-6p952\" (UID: \"867b0a1e-9249-4fbc-be63-e0495f3aff8b\") " pod="opendatahub/opendatahub-operator-controller-manager-8699bb9d7c-6p952" Jul 28 01:20:37.463737 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:37.463711 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/867b0a1e-9249-4fbc-be63-e0495f3aff8b-manifests\") pod \"opendatahub-operator-controller-manager-8699bb9d7c-6p952\" (UID: \"867b0a1e-9249-4fbc-be63-e0495f3aff8b\") " pod="opendatahub/opendatahub-operator-controller-manager-8699bb9d7c-6p952" Jul 28 01:20:37.564913 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:37.564882 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/867b0a1e-9249-4fbc-be63-e0495f3aff8b-webhook-cert\") pod \"opendatahub-operator-controller-manager-8699bb9d7c-6p952\" (UID: \"867b0a1e-9249-4fbc-be63-e0495f3aff8b\") " pod="opendatahub/opendatahub-operator-controller-manager-8699bb9d7c-6p952" Jul 28 01:20:37.565090 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:37.564918 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/867b0a1e-9249-4fbc-be63-e0495f3aff8b-manifests\") pod \"opendatahub-operator-controller-manager-8699bb9d7c-6p952\" (UID: \"867b0a1e-9249-4fbc-be63-e0495f3aff8b\") " pod="opendatahub/opendatahub-operator-controller-manager-8699bb9d7c-6p952" Jul 28 01:20:37.565090 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:37.564966 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7fwz\" (UniqueName: \"kubernetes.io/projected/867b0a1e-9249-4fbc-be63-e0495f3aff8b-kube-api-access-p7fwz\") pod \"opendatahub-operator-controller-manager-8699bb9d7c-6p952\" (UID: \"867b0a1e-9249-4fbc-be63-e0495f3aff8b\") " pod="opendatahub/opendatahub-operator-controller-manager-8699bb9d7c-6p952" Jul 28 01:20:37.565090 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:37.564991 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/867b0a1e-9249-4fbc-be63-e0495f3aff8b-tmp\") pod \"opendatahub-operator-controller-manager-8699bb9d7c-6p952\" (UID: \"867b0a1e-9249-4fbc-be63-e0495f3aff8b\") " pod="opendatahub/opendatahub-operator-controller-manager-8699bb9d7c-6p952" Jul 28 01:20:37.565090 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:37.565020 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/867b0a1e-9249-4fbc-be63-e0495f3aff8b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8699bb9d7c-6p952\" (UID: \"867b0a1e-9249-4fbc-be63-e0495f3aff8b\") " pod="opendatahub/opendatahub-operator-controller-manager-8699bb9d7c-6p952" Jul 28 01:20:37.565458 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:37.565433 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/867b0a1e-9249-4fbc-be63-e0495f3aff8b-manifests\") pod \"opendatahub-operator-controller-manager-8699bb9d7c-6p952\" (UID: \"867b0a1e-9249-4fbc-be63-e0495f3aff8b\") " pod="opendatahub/opendatahub-operator-controller-manager-8699bb9d7c-6p952" Jul 28 01:20:37.567287 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:37.567258 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/867b0a1e-9249-4fbc-be63-e0495f3aff8b-tmp\") pod \"opendatahub-operator-controller-manager-8699bb9d7c-6p952\" (UID: \"867b0a1e-9249-4fbc-be63-e0495f3aff8b\") " pod="opendatahub/opendatahub-operator-controller-manager-8699bb9d7c-6p952" Jul 28 01:20:37.567377 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:37.567348 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/867b0a1e-9249-4fbc-be63-e0495f3aff8b-webhook-cert\") pod \"opendatahub-operator-controller-manager-8699bb9d7c-6p952\" (UID: \"867b0a1e-9249-4fbc-be63-e0495f3aff8b\") " pod="opendatahub/opendatahub-operator-controller-manager-8699bb9d7c-6p952" Jul 28 01:20:37.567418 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:37.567380 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/867b0a1e-9249-4fbc-be63-e0495f3aff8b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8699bb9d7c-6p952\" (UID: \"867b0a1e-9249-4fbc-be63-e0495f3aff8b\") " pod="opendatahub/opendatahub-operator-controller-manager-8699bb9d7c-6p952" Jul 28 01:20:37.572837 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:37.572814 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7fwz\" (UniqueName: \"kubernetes.io/projected/867b0a1e-9249-4fbc-be63-e0495f3aff8b-kube-api-access-p7fwz\") pod \"opendatahub-operator-controller-manager-8699bb9d7c-6p952\" (UID: \"867b0a1e-9249-4fbc-be63-e0495f3aff8b\") " pod="opendatahub/opendatahub-operator-controller-manager-8699bb9d7c-6p952" Jul 28 01:20:37.715871 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:37.715832 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8699bb9d7c-6p952" Jul 28 01:20:37.852952 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:37.852917 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8699bb9d7c-6p952"] Jul 28 01:20:37.856978 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:20:37.856948 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod867b0a1e_9249_4fbc_be63_e0495f3aff8b.slice/crio-32957df042329ffcfdb210b22326c510e279ca12caa77d4b2d0cc2dc270c91aa WatchSource:0}: Error finding container 32957df042329ffcfdb210b22326c510e279ca12caa77d4b2d0cc2dc270c91aa: Status 404 returned error can't find the container with id 32957df042329ffcfdb210b22326c510e279ca12caa77d4b2d0cc2dc270c91aa Jul 28 01:20:38.335678 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:38.335645 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8699bb9d7c-6p952" event={"ID":"867b0a1e-9249-4fbc-be63-e0495f3aff8b","Type":"ContainerStarted","Data":"32957df042329ffcfdb210b22326c510e279ca12caa77d4b2d0cc2dc270c91aa"} Jul 28 01:20:41.350998 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:41.350956 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8699bb9d7c-6p952" event={"ID":"867b0a1e-9249-4fbc-be63-e0495f3aff8b","Type":"ContainerStarted","Data":"c56dc6da5c51e03b5b056dc57c9b37157f32b8867b60148669244831a4785e7c"} Jul 28 01:20:42.355992 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:42.355955 2565 generic.go:356] "Generic (PLEG): container finished" podID="867b0a1e-9249-4fbc-be63-e0495f3aff8b" containerID="c56dc6da5c51e03b5b056dc57c9b37157f32b8867b60148669244831a4785e7c" exitCode=0 Jul 28 01:20:42.356421 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:42.356004 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8699bb9d7c-6p952" event={"ID":"867b0a1e-9249-4fbc-be63-e0495f3aff8b","Type":"ContainerDied","Data":"c56dc6da5c51e03b5b056dc57c9b37157f32b8867b60148669244831a4785e7c"} Jul 28 01:20:42.951919 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:42.951895 2565 kubelet.go:1648] "Image garbage collection succeeded" Jul 28 01:20:43.362473 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:43.362385 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8699bb9d7c-6p952" event={"ID":"867b0a1e-9249-4fbc-be63-e0495f3aff8b","Type":"ContainerStarted","Data":"cb60c3b1b70c0df4e799c6bde2dd9cfe5d4ff5eba0e5593ac7c091248f82bf03"} Jul 28 01:20:43.363959 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:43.362535 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-8699bb9d7c-6p952" Jul 28 01:20:43.384429 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:43.384374 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-8699bb9d7c-6p952" podStartSLOduration=2.179493508 podStartE2EDuration="6.384354783s" totalImagesPullingTime="3.155325049s" totalInitContainerRuntime="1.049536226s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:20:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:20:43.384114787 +0000 UTC m=+300.818513103" watchObservedRunningTime="2026-07-28 01:20:43.384354783 +0000 UTC m=+300.818753100" Jul 28 01:20:54.368462 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:54.368425 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-8699bb9d7c-6p952" Jul 28 01:20:54.619419 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:54.619342 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7d7cccf554-p9gjs"] Jul 28 01:20:54.628914 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:54.628885 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7d7cccf554-p9gjs" Jul 28 01:20:54.632130 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:54.632101 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 28 01:20:54.632130 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:54.632122 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 28 01:20:54.632340 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:54.632109 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mzx7x\"" Jul 28 01:20:54.632340 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:54.632102 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 28 01:20:54.632340 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:54.632284 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 28 01:20:54.632340 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:54.632326 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7d7cccf554-p9gjs"] Jul 28 01:20:54.632548 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:54.632401 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 28 01:20:54.700542 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:54.700505 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f3cab44-ce9b-4d19-82ec-d47a0876b021-cert\") pod \"lws-controller-manager-7d7cccf554-p9gjs\" (UID: \"9f3cab44-ce9b-4d19-82ec-d47a0876b021\") " pod="openshift-lws-operator/lws-controller-manager-7d7cccf554-p9gjs" Jul 28 01:20:54.700710 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:54.700547 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qknp8\" (UniqueName: \"kubernetes.io/projected/9f3cab44-ce9b-4d19-82ec-d47a0876b021-kube-api-access-qknp8\") pod \"lws-controller-manager-7d7cccf554-p9gjs\" (UID: \"9f3cab44-ce9b-4d19-82ec-d47a0876b021\") " pod="openshift-lws-operator/lws-controller-manager-7d7cccf554-p9gjs" Jul 28 01:20:54.700710 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:54.700578 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9f3cab44-ce9b-4d19-82ec-d47a0876b021-metrics-cert\") pod \"lws-controller-manager-7d7cccf554-p9gjs\" (UID: \"9f3cab44-ce9b-4d19-82ec-d47a0876b021\") " pod="openshift-lws-operator/lws-controller-manager-7d7cccf554-p9gjs" Jul 28 01:20:54.700710 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:54.700672 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9f3cab44-ce9b-4d19-82ec-d47a0876b021-manager-config\") pod \"lws-controller-manager-7d7cccf554-p9gjs\" (UID: \"9f3cab44-ce9b-4d19-82ec-d47a0876b021\") " pod="openshift-lws-operator/lws-controller-manager-7d7cccf554-p9gjs" Jul 28 01:20:54.801303 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:54.801267 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qknp8\" (UniqueName: \"kubernetes.io/projected/9f3cab44-ce9b-4d19-82ec-d47a0876b021-kube-api-access-qknp8\") pod \"lws-controller-manager-7d7cccf554-p9gjs\" (UID: \"9f3cab44-ce9b-4d19-82ec-d47a0876b021\") " pod="openshift-lws-operator/lws-controller-manager-7d7cccf554-p9gjs" Jul 28 01:20:54.801439 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:54.801384 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9f3cab44-ce9b-4d19-82ec-d47a0876b021-metrics-cert\") pod \"lws-controller-manager-7d7cccf554-p9gjs\" (UID: \"9f3cab44-ce9b-4d19-82ec-d47a0876b021\") " pod="openshift-lws-operator/lws-controller-manager-7d7cccf554-p9gjs" Jul 28 01:20:54.801439 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:54.801424 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9f3cab44-ce9b-4d19-82ec-d47a0876b021-manager-config\") pod \"lws-controller-manager-7d7cccf554-p9gjs\" (UID: \"9f3cab44-ce9b-4d19-82ec-d47a0876b021\") " pod="openshift-lws-operator/lws-controller-manager-7d7cccf554-p9gjs" Jul 28 01:20:54.801506 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:54.801479 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f3cab44-ce9b-4d19-82ec-d47a0876b021-cert\") pod \"lws-controller-manager-7d7cccf554-p9gjs\" (UID: \"9f3cab44-ce9b-4d19-82ec-d47a0876b021\") " pod="openshift-lws-operator/lws-controller-manager-7d7cccf554-p9gjs" Jul 28 01:20:54.802254 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:54.802229 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9f3cab44-ce9b-4d19-82ec-d47a0876b021-manager-config\") pod \"lws-controller-manager-7d7cccf554-p9gjs\" (UID: \"9f3cab44-ce9b-4d19-82ec-d47a0876b021\") " pod="openshift-lws-operator/lws-controller-manager-7d7cccf554-p9gjs" Jul 28 01:20:54.803959 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:54.803932 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9f3cab44-ce9b-4d19-82ec-d47a0876b021-metrics-cert\") pod \"lws-controller-manager-7d7cccf554-p9gjs\" (UID: \"9f3cab44-ce9b-4d19-82ec-d47a0876b021\") " pod="openshift-lws-operator/lws-controller-manager-7d7cccf554-p9gjs" Jul 28 01:20:54.804077 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:54.803962 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f3cab44-ce9b-4d19-82ec-d47a0876b021-cert\") pod \"lws-controller-manager-7d7cccf554-p9gjs\" (UID: \"9f3cab44-ce9b-4d19-82ec-d47a0876b021\") " pod="openshift-lws-operator/lws-controller-manager-7d7cccf554-p9gjs" Jul 28 01:20:54.810070 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:54.810026 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qknp8\" (UniqueName: \"kubernetes.io/projected/9f3cab44-ce9b-4d19-82ec-d47a0876b021-kube-api-access-qknp8\") pod \"lws-controller-manager-7d7cccf554-p9gjs\" (UID: \"9f3cab44-ce9b-4d19-82ec-d47a0876b021\") " pod="openshift-lws-operator/lws-controller-manager-7d7cccf554-p9gjs" Jul 28 01:20:54.938965 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:54.938930 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7d7cccf554-p9gjs" Jul 28 01:20:55.068438 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:55.068342 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7d7cccf554-p9gjs"] Jul 28 01:20:55.071520 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:20:55.071483 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f3cab44_ce9b_4d19_82ec_d47a0876b021.slice/crio-812b3958ae5d71a6dbcf0953df66f9071f9000f4709d700288bf1a19233adf47 WatchSource:0}: Error finding container 812b3958ae5d71a6dbcf0953df66f9071f9000f4709d700288bf1a19233adf47: Status 404 returned error can't find the container with id 812b3958ae5d71a6dbcf0953df66f9071f9000f4709d700288bf1a19233adf47 Jul 28 01:20:55.073878 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:55.073864 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 01:20:55.413143 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:55.413060 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7d7cccf554-p9gjs" event={"ID":"9f3cab44-ce9b-4d19-82ec-d47a0876b021","Type":"ContainerStarted","Data":"812b3958ae5d71a6dbcf0953df66f9071f9000f4709d700288bf1a19233adf47"} Jul 28 01:20:57.438137 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:57.438100 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7d7cccf554-p9gjs" event={"ID":"9f3cab44-ce9b-4d19-82ec-d47a0876b021","Type":"ContainerStarted","Data":"f03781cc5a8020c723bbd460cc9c0e384efea1ebffc9c413e283e830cbc6cb78"} Jul 28 01:20:57.438621 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:57.438253 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7d7cccf554-p9gjs" Jul 28 01:20:57.454654 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:20:57.454607 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7d7cccf554-p9gjs" podStartSLOduration=1.725105278 podStartE2EDuration="3.454592211s" totalImagesPullingTime="1.729486933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:20:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:20:57.454230245 +0000 UTC m=+314.888628571" watchObservedRunningTime="2026-07-28 01:20:57.454592211 +0000 UTC m=+314.888990524" Jul 28 01:21:04.890087 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:04.890045 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-85hcz"] Jul 28 01:21:04.894633 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:04.894599 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-85hcz" Jul 28 01:21:04.897316 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:04.897292 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 28 01:21:04.898451 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:04.898432 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 28 01:21:04.898642 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:04.898491 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 28 01:21:04.898735 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:04.898509 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 28 01:21:04.898857 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:04.898536 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 28 01:21:04.899002 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:04.898979 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-cbtrp\"" Jul 28 01:21:04.899200 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:04.899183 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 28 01:21:04.908272 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:04.908239 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-85hcz"] Jul 28 01:21:04.998642 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:04.998604 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/15fcf533-e2cd-4e92-8891-1d27b46f6ea9-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-85hcz\" (UID: \"15fcf533-e2cd-4e92-8891-1d27b46f6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-85hcz" Jul 28 01:21:04.998817 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:04.998657 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/15fcf533-e2cd-4e92-8891-1d27b46f6ea9-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-85hcz\" (UID: \"15fcf533-e2cd-4e92-8891-1d27b46f6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-85hcz" Jul 28 01:21:04.998817 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:04.998683 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/15fcf533-e2cd-4e92-8891-1d27b46f6ea9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-85hcz\" (UID: \"15fcf533-e2cd-4e92-8891-1d27b46f6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-85hcz" Jul 28 01:21:04.998817 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:04.998708 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kjfs\" (UniqueName: \"kubernetes.io/projected/15fcf533-e2cd-4e92-8891-1d27b46f6ea9-kube-api-access-7kjfs\") pod \"istiod-openshift-gateway-5677ccb49b-85hcz\" (UID: \"15fcf533-e2cd-4e92-8891-1d27b46f6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-85hcz" Jul 28 01:21:04.998923 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:04.998810 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/15fcf533-e2cd-4e92-8891-1d27b46f6ea9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-85hcz\" (UID: \"15fcf533-e2cd-4e92-8891-1d27b46f6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-85hcz" Jul 28 01:21:04.998923 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:04.998845 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/15fcf533-e2cd-4e92-8891-1d27b46f6ea9-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-85hcz\" (UID: \"15fcf533-e2cd-4e92-8891-1d27b46f6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-85hcz" Jul 28 01:21:04.998923 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:04.998866 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15fcf533-e2cd-4e92-8891-1d27b46f6ea9-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-85hcz\" (UID: \"15fcf533-e2cd-4e92-8891-1d27b46f6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-85hcz" Jul 28 01:21:05.099719 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:05.099688 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/15fcf533-e2cd-4e92-8891-1d27b46f6ea9-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-85hcz\" (UID: \"15fcf533-e2cd-4e92-8891-1d27b46f6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-85hcz" Jul 28 01:21:05.099719 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:05.099723 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/15fcf533-e2cd-4e92-8891-1d27b46f6ea9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-85hcz\" (UID: \"15fcf533-e2cd-4e92-8891-1d27b46f6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-85hcz" Jul 28 01:21:05.099976 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:05.099739 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kjfs\" (UniqueName: \"kubernetes.io/projected/15fcf533-e2cd-4e92-8891-1d27b46f6ea9-kube-api-access-7kjfs\") pod \"istiod-openshift-gateway-5677ccb49b-85hcz\" (UID: \"15fcf533-e2cd-4e92-8891-1d27b46f6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-85hcz" Jul 28 01:21:05.099976 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:05.099794 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/15fcf533-e2cd-4e92-8891-1d27b46f6ea9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-85hcz\" (UID: \"15fcf533-e2cd-4e92-8891-1d27b46f6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-85hcz" Jul 28 01:21:05.099976 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:05.099811 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/15fcf533-e2cd-4e92-8891-1d27b46f6ea9-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-85hcz\" (UID: \"15fcf533-e2cd-4e92-8891-1d27b46f6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-85hcz" Jul 28 01:21:05.099976 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:05.099827 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15fcf533-e2cd-4e92-8891-1d27b46f6ea9-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-85hcz\" (UID: \"15fcf533-e2cd-4e92-8891-1d27b46f6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-85hcz" Jul 28 01:21:05.099976 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:05.099854 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/15fcf533-e2cd-4e92-8891-1d27b46f6ea9-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-85hcz\" (UID: \"15fcf533-e2cd-4e92-8891-1d27b46f6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-85hcz" Jul 28 01:21:05.100407 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:05.100385 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/15fcf533-e2cd-4e92-8891-1d27b46f6ea9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-85hcz\" (UID: \"15fcf533-e2cd-4e92-8891-1d27b46f6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-85hcz" Jul 28 01:21:05.102169 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:05.102146 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/15fcf533-e2cd-4e92-8891-1d27b46f6ea9-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-85hcz\" (UID: \"15fcf533-e2cd-4e92-8891-1d27b46f6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-85hcz" Jul 28 01:21:05.102169 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:05.102165 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/15fcf533-e2cd-4e92-8891-1d27b46f6ea9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-85hcz\" (UID: \"15fcf533-e2cd-4e92-8891-1d27b46f6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-85hcz" Jul 28 01:21:05.102299 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:05.102240 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/15fcf533-e2cd-4e92-8891-1d27b46f6ea9-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-85hcz\" (UID: \"15fcf533-e2cd-4e92-8891-1d27b46f6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-85hcz" Jul 28 01:21:05.102368 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:05.102350 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/15fcf533-e2cd-4e92-8891-1d27b46f6ea9-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-85hcz\" (UID: \"15fcf533-e2cd-4e92-8891-1d27b46f6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-85hcz" Jul 28 01:21:05.108483 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:05.108459 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15fcf533-e2cd-4e92-8891-1d27b46f6ea9-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-85hcz\" (UID: \"15fcf533-e2cd-4e92-8891-1d27b46f6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-85hcz" Jul 28 01:21:05.108572 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:05.108504 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kjfs\" (UniqueName: \"kubernetes.io/projected/15fcf533-e2cd-4e92-8891-1d27b46f6ea9-kube-api-access-7kjfs\") pod \"istiod-openshift-gateway-5677ccb49b-85hcz\" (UID: \"15fcf533-e2cd-4e92-8891-1d27b46f6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-85hcz" Jul 28 01:21:05.206105 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:05.206073 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-85hcz" Jul 28 01:21:05.338637 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:05.338603 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-85hcz"] Jul 28 01:21:05.340655 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:21:05.340632 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15fcf533_e2cd_4e92_8891_1d27b46f6ea9.slice/crio-147af6d5308915e3efc5514371c99c766b766262b0ecec3b44ce61532319c423 WatchSource:0}: Error finding container 147af6d5308915e3efc5514371c99c766b766262b0ecec3b44ce61532319c423: Status 404 returned error can't find the container with id 147af6d5308915e3efc5514371c99c766b766262b0ecec3b44ce61532319c423 Jul 28 01:21:05.472354 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:05.472283 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-85hcz" event={"ID":"15fcf533-e2cd-4e92-8891-1d27b46f6ea9","Type":"ContainerStarted","Data":"147af6d5308915e3efc5514371c99c766b766262b0ecec3b44ce61532319c423"} Jul 28 01:21:07.881639 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:07.881595 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:21:07.882001 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:07.881680 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:21:08.444817 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.444789 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7d7cccf554-p9gjs" Jul 28 01:21:08.489300 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.489258 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-85hcz" event={"ID":"15fcf533-e2cd-4e92-8891-1d27b46f6ea9","Type":"ContainerStarted","Data":"270d55a380179e36e4263384387909863f05d883cbfa8f00aa7fe7f0578ce561"} Jul 28 01:21:08.489529 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.489502 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-85hcz" Jul 28 01:21:08.491266 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.491236 2565 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-85hcz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 28 01:21:08.491379 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.491294 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-85hcz" podUID="15fcf533-e2cd-4e92-8891-1d27b46f6ea9" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 01:21:08.516822 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.516788 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj"] Jul 28 01:21:08.520592 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.520569 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj" Jul 28 01:21:08.528005 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.527973 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-ggqfx\"" Jul 28 01:21:08.544844 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.544817 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj"] Jul 28 01:21:08.548839 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.548796 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-85hcz" podStartSLOduration=2.0101339 podStartE2EDuration="4.54878015s" totalImagesPullingTime="2.53864625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:21:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:21:08.547135355 +0000 UTC m=+325.981533670" watchObservedRunningTime="2026-07-28 01:21:08.54878015 +0000 UTC m=+325.983178464" Jul 28 01:21:08.632899 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.632863 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d56a28fd-a428-4306-95a9-065e8e92e8bc-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj\" (UID: \"d56a28fd-a428-4306-95a9-065e8e92e8bc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj" Jul 28 01:21:08.632899 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.632901 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d56a28fd-a428-4306-95a9-065e8e92e8bc-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj\" (UID: \"d56a28fd-a428-4306-95a9-065e8e92e8bc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj" Jul 28 01:21:08.633175 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.632930 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d56a28fd-a428-4306-95a9-065e8e92e8bc-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj\" (UID: \"d56a28fd-a428-4306-95a9-065e8e92e8bc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj" Jul 28 01:21:08.633175 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.632975 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d56a28fd-a428-4306-95a9-065e8e92e8bc-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj\" (UID: \"d56a28fd-a428-4306-95a9-065e8e92e8bc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj" Jul 28 01:21:08.633175 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.633096 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d56a28fd-a428-4306-95a9-065e8e92e8bc-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj\" (UID: \"d56a28fd-a428-4306-95a9-065e8e92e8bc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj" Jul 28 01:21:08.633175 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.633123 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2n5b\" (UniqueName: \"kubernetes.io/projected/d56a28fd-a428-4306-95a9-065e8e92e8bc-kube-api-access-t2n5b\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj\" (UID: \"d56a28fd-a428-4306-95a9-065e8e92e8bc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj" Jul 28 01:21:08.633175 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.633161 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56a28fd-a428-4306-95a9-065e8e92e8bc-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj\" (UID: \"d56a28fd-a428-4306-95a9-065e8e92e8bc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj" Jul 28 01:21:08.633427 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.633189 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d56a28fd-a428-4306-95a9-065e8e92e8bc-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj\" (UID: \"d56a28fd-a428-4306-95a9-065e8e92e8bc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj" Jul 28 01:21:08.633427 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.633243 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56a28fd-a428-4306-95a9-065e8e92e8bc-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj\" (UID: \"d56a28fd-a428-4306-95a9-065e8e92e8bc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj" Jul 28 01:21:08.734012 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.733931 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d56a28fd-a428-4306-95a9-065e8e92e8bc-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj\" (UID: \"d56a28fd-a428-4306-95a9-065e8e92e8bc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj" Jul 28 01:21:08.734012 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.733966 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d56a28fd-a428-4306-95a9-065e8e92e8bc-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj\" (UID: \"d56a28fd-a428-4306-95a9-065e8e92e8bc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj" Jul 28 01:21:08.734012 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.733987 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d56a28fd-a428-4306-95a9-065e8e92e8bc-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj\" (UID: \"d56a28fd-a428-4306-95a9-065e8e92e8bc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj" Jul 28 01:21:08.734012 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.734006 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d56a28fd-a428-4306-95a9-065e8e92e8bc-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj\" (UID: \"d56a28fd-a428-4306-95a9-065e8e92e8bc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj" Jul 28 01:21:08.734356 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.734051 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d56a28fd-a428-4306-95a9-065e8e92e8bc-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj\" (UID: \"d56a28fd-a428-4306-95a9-065e8e92e8bc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj" Jul 28 01:21:08.734356 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.734077 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2n5b\" (UniqueName: \"kubernetes.io/projected/d56a28fd-a428-4306-95a9-065e8e92e8bc-kube-api-access-t2n5b\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj\" (UID: \"d56a28fd-a428-4306-95a9-065e8e92e8bc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj" Jul 28 01:21:08.734356 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.734112 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56a28fd-a428-4306-95a9-065e8e92e8bc-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj\" (UID: \"d56a28fd-a428-4306-95a9-065e8e92e8bc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj" Jul 28 01:21:08.734356 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.734139 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d56a28fd-a428-4306-95a9-065e8e92e8bc-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj\" (UID: \"d56a28fd-a428-4306-95a9-065e8e92e8bc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj" Jul 28 01:21:08.734356 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.734169 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56a28fd-a428-4306-95a9-065e8e92e8bc-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj\" (UID: \"d56a28fd-a428-4306-95a9-065e8e92e8bc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj" Jul 28 01:21:08.734590 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.734457 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d56a28fd-a428-4306-95a9-065e8e92e8bc-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj\" (UID: \"d56a28fd-a428-4306-95a9-065e8e92e8bc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj" Jul 28 01:21:08.734590 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.734497 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d56a28fd-a428-4306-95a9-065e8e92e8bc-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj\" (UID: \"d56a28fd-a428-4306-95a9-065e8e92e8bc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj" Jul 28 01:21:08.734699 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.734683 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56a28fd-a428-4306-95a9-065e8e92e8bc-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj\" (UID: \"d56a28fd-a428-4306-95a9-065e8e92e8bc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj" Jul 28 01:21:08.734784 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.734746 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56a28fd-a428-4306-95a9-065e8e92e8bc-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj\" (UID: \"d56a28fd-a428-4306-95a9-065e8e92e8bc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj" Jul 28 01:21:08.735232 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.735204 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d56a28fd-a428-4306-95a9-065e8e92e8bc-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj\" (UID: \"d56a28fd-a428-4306-95a9-065e8e92e8bc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj" Jul 28 01:21:08.736928 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.736900 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d56a28fd-a428-4306-95a9-065e8e92e8bc-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj\" (UID: \"d56a28fd-a428-4306-95a9-065e8e92e8bc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj" Jul 28 01:21:08.737011 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.736959 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d56a28fd-a428-4306-95a9-065e8e92e8bc-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj\" (UID: \"d56a28fd-a428-4306-95a9-065e8e92e8bc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj" Jul 28 01:21:08.743696 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.743675 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d56a28fd-a428-4306-95a9-065e8e92e8bc-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj\" (UID: \"d56a28fd-a428-4306-95a9-065e8e92e8bc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj" Jul 28 01:21:08.747793 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.747773 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2n5b\" (UniqueName: \"kubernetes.io/projected/d56a28fd-a428-4306-95a9-065e8e92e8bc-kube-api-access-t2n5b\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj\" (UID: \"d56a28fd-a428-4306-95a9-065e8e92e8bc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj" Jul 28 01:21:08.833403 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.833360 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj" Jul 28 01:21:08.969593 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:08.969566 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj"] Jul 28 01:21:08.971670 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:21:08.971647 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56a28fd_a428_4306_95a9_065e8e92e8bc.slice/crio-78d5504657f2b118bb5216bfd168a1703dddb5fd2fab5ec523d3431cbd2310aa WatchSource:0}: Error finding container 78d5504657f2b118bb5216bfd168a1703dddb5fd2fab5ec523d3431cbd2310aa: Status 404 returned error can't find the container with id 78d5504657f2b118bb5216bfd168a1703dddb5fd2fab5ec523d3431cbd2310aa Jul 28 01:21:09.494760 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:09.494726 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj" event={"ID":"d56a28fd-a428-4306-95a9-065e8e92e8bc","Type":"ContainerStarted","Data":"78d5504657f2b118bb5216bfd168a1703dddb5fd2fab5ec523d3431cbd2310aa"} Jul 28 01:21:09.495492 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:09.495473 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-85hcz" Jul 28 01:21:13.640299 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:13.640262 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:21:13.640591 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:13.640331 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:21:13.640591 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:13.640360 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:21:14.518492 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:14.518456 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj" event={"ID":"d56a28fd-a428-4306-95a9-065e8e92e8bc","Type":"ContainerStarted","Data":"dc9d8de4dda7b894546e67dab745ec44d3f117644531c4d5aa01ce6bde20f168"} Jul 28 01:21:14.559194 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:14.559142 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj" podStartSLOduration=1.893405676 podStartE2EDuration="6.559128271s" totalImagesPullingTime="4.665722595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:21:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:21:14.55897543 +0000 UTC m=+331.993373745" watchObservedRunningTime="2026-07-28 01:21:14.559128271 +0000 UTC m=+331.993526585" Jul 28 01:21:14.834086 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:14.833987 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj" Jul 28 01:21:14.838712 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:14.838686 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj" Jul 28 01:21:15.522561 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:15.522529 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj" Jul 28 01:21:15.523491 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:21:15.523467 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj" Jul 28 01:22:13.736466 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:13.736429 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g7xcs"] Jul 28 01:22:13.738934 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:13.738917 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g7xcs" Jul 28 01:22:13.741795 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:13.741769 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xgtzm\"" Jul 28 01:22:13.741795 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:13.741791 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 28 01:22:13.741948 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:13.741816 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 28 01:22:13.747488 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:13.747468 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g7xcs"] Jul 28 01:22:13.890547 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:13.890516 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ce1d280-4f01-4cae-8d9b-57d48994c94b-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g7xcs\" (UID: \"2ce1d280-4f01-4cae-8d9b-57d48994c94b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g7xcs" Jul 28 01:22:13.890720 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:13.890554 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smc7l\" (UniqueName: \"kubernetes.io/projected/2ce1d280-4f01-4cae-8d9b-57d48994c94b-kube-api-access-smc7l\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g7xcs\" (UID: \"2ce1d280-4f01-4cae-8d9b-57d48994c94b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g7xcs" Jul 28 01:22:13.890720 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:13.890657 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ce1d280-4f01-4cae-8d9b-57d48994c94b-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g7xcs\" (UID: \"2ce1d280-4f01-4cae-8d9b-57d48994c94b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g7xcs" Jul 28 01:22:13.991652 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:13.991576 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ce1d280-4f01-4cae-8d9b-57d48994c94b-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g7xcs\" (UID: \"2ce1d280-4f01-4cae-8d9b-57d48994c94b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g7xcs" Jul 28 01:22:13.991652 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:13.991612 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smc7l\" (UniqueName: \"kubernetes.io/projected/2ce1d280-4f01-4cae-8d9b-57d48994c94b-kube-api-access-smc7l\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g7xcs\" (UID: \"2ce1d280-4f01-4cae-8d9b-57d48994c94b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g7xcs" Jul 28 01:22:13.991827 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:13.991806 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ce1d280-4f01-4cae-8d9b-57d48994c94b-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g7xcs\" (UID: \"2ce1d280-4f01-4cae-8d9b-57d48994c94b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g7xcs" Jul 28 01:22:13.992052 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:13.992016 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ce1d280-4f01-4cae-8d9b-57d48994c94b-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g7xcs\" (UID: \"2ce1d280-4f01-4cae-8d9b-57d48994c94b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g7xcs" Jul 28 01:22:13.992128 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:13.992112 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ce1d280-4f01-4cae-8d9b-57d48994c94b-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g7xcs\" (UID: \"2ce1d280-4f01-4cae-8d9b-57d48994c94b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g7xcs" Jul 28 01:22:14.000118 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:14.000082 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smc7l\" (UniqueName: \"kubernetes.io/projected/2ce1d280-4f01-4cae-8d9b-57d48994c94b-kube-api-access-smc7l\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g7xcs\" (UID: \"2ce1d280-4f01-4cae-8d9b-57d48994c94b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g7xcs" Jul 28 01:22:14.048951 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:14.048924 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g7xcs" Jul 28 01:22:14.172774 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:14.172726 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g7xcs"] Jul 28 01:22:14.175104 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:22:14.175067 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ce1d280_4f01_4cae_8d9b_57d48994c94b.slice/crio-da73b2585fbaf5c1c236def5789d414d7ddd13d78004cdcb3e96c401574527e9 WatchSource:0}: Error finding container da73b2585fbaf5c1c236def5789d414d7ddd13d78004cdcb3e96c401574527e9: Status 404 returned error can't find the container with id da73b2585fbaf5c1c236def5789d414d7ddd13d78004cdcb3e96c401574527e9 Jul 28 01:22:14.350804 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:14.350775 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tptfg"] Jul 28 01:22:14.353208 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:14.353187 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tptfg" Jul 28 01:22:14.362495 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:14.362473 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tptfg"] Jul 28 01:22:14.496596 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:14.496567 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb1f9dce-dee3-446c-a9bf-16764b20a710-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tptfg\" (UID: \"bb1f9dce-dee3-446c-a9bf-16764b20a710\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tptfg" Jul 28 01:22:14.496726 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:14.496605 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb1f9dce-dee3-446c-a9bf-16764b20a710-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tptfg\" (UID: \"bb1f9dce-dee3-446c-a9bf-16764b20a710\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tptfg" Jul 28 01:22:14.496726 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:14.496702 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq5n2\" (UniqueName: \"kubernetes.io/projected/bb1f9dce-dee3-446c-a9bf-16764b20a710-kube-api-access-gq5n2\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tptfg\" (UID: \"bb1f9dce-dee3-446c-a9bf-16764b20a710\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tptfg" Jul 28 01:22:14.597851 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:14.597771 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq5n2\" (UniqueName: \"kubernetes.io/projected/bb1f9dce-dee3-446c-a9bf-16764b20a710-kube-api-access-gq5n2\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tptfg\" (UID: \"bb1f9dce-dee3-446c-a9bf-16764b20a710\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tptfg" Jul 28 01:22:14.597851 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:14.597834 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb1f9dce-dee3-446c-a9bf-16764b20a710-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tptfg\" (UID: \"bb1f9dce-dee3-446c-a9bf-16764b20a710\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tptfg" Jul 28 01:22:14.598060 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:14.597864 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb1f9dce-dee3-446c-a9bf-16764b20a710-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tptfg\" (UID: \"bb1f9dce-dee3-446c-a9bf-16764b20a710\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tptfg" Jul 28 01:22:14.598249 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:14.598226 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb1f9dce-dee3-446c-a9bf-16764b20a710-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tptfg\" (UID: \"bb1f9dce-dee3-446c-a9bf-16764b20a710\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tptfg" Jul 28 01:22:14.598316 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:14.598261 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb1f9dce-dee3-446c-a9bf-16764b20a710-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tptfg\" (UID: \"bb1f9dce-dee3-446c-a9bf-16764b20a710\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tptfg" Jul 28 01:22:14.607647 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:14.607628 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq5n2\" (UniqueName: \"kubernetes.io/projected/bb1f9dce-dee3-446c-a9bf-16764b20a710-kube-api-access-gq5n2\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tptfg\" (UID: \"bb1f9dce-dee3-446c-a9bf-16764b20a710\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tptfg" Jul 28 01:22:14.663985 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:14.663955 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tptfg" Jul 28 01:22:14.783166 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:14.783027 2565 generic.go:356] "Generic (PLEG): container finished" podID="2ce1d280-4f01-4cae-8d9b-57d48994c94b" containerID="9af9cc1706d10a2076f7b784880c51b7feeb582e8439cfcb359dfae026aebf5e" exitCode=0 Jul 28 01:22:14.783166 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:14.783133 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g7xcs" event={"ID":"2ce1d280-4f01-4cae-8d9b-57d48994c94b","Type":"ContainerDied","Data":"9af9cc1706d10a2076f7b784880c51b7feeb582e8439cfcb359dfae026aebf5e"} Jul 28 01:22:14.783691 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:14.783212 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g7xcs" event={"ID":"2ce1d280-4f01-4cae-8d9b-57d48994c94b","Type":"ContainerStarted","Data":"da73b2585fbaf5c1c236def5789d414d7ddd13d78004cdcb3e96c401574527e9"} Jul 28 01:22:14.785537 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:14.785515 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tptfg"] Jul 28 01:22:14.788227 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:22:14.788203 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb1f9dce_dee3_446c_a9bf_16764b20a710.slice/crio-b476a33ae8803c7d3f233f69b29bc463aa35a3361c3d6f023f533c7f9672a411 WatchSource:0}: Error finding container b476a33ae8803c7d3f233f69b29bc463aa35a3361c3d6f023f533c7f9672a411: Status 404 returned error can't find the container with id b476a33ae8803c7d3f233f69b29bc463aa35a3361c3d6f023f533c7f9672a411 Jul 28 01:22:14.938155 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:14.938128 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tzrmz"] Jul 28 01:22:14.940486 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:14.940465 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tzrmz" Jul 28 01:22:14.949286 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:14.949264 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tzrmz"] Jul 28 01:22:15.000557 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:15.000530 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53045446-768f-47e4-9ef4-5eda1c261514-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tzrmz\" (UID: \"53045446-768f-47e4-9ef4-5eda1c261514\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tzrmz" Jul 28 01:22:15.000702 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:15.000579 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53045446-768f-47e4-9ef4-5eda1c261514-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tzrmz\" (UID: \"53045446-768f-47e4-9ef4-5eda1c261514\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tzrmz" Jul 28 01:22:15.000702 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:15.000620 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp4k4\" (UniqueName: \"kubernetes.io/projected/53045446-768f-47e4-9ef4-5eda1c261514-kube-api-access-vp4k4\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tzrmz\" (UID: \"53045446-768f-47e4-9ef4-5eda1c261514\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tzrmz" Jul 28 01:22:15.101651 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:15.101626 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53045446-768f-47e4-9ef4-5eda1c261514-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tzrmz\" (UID: \"53045446-768f-47e4-9ef4-5eda1c261514\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tzrmz" Jul 28 01:22:15.101777 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:15.101660 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53045446-768f-47e4-9ef4-5eda1c261514-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tzrmz\" (UID: \"53045446-768f-47e4-9ef4-5eda1c261514\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tzrmz" Jul 28 01:22:15.101777 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:15.101683 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp4k4\" (UniqueName: \"kubernetes.io/projected/53045446-768f-47e4-9ef4-5eda1c261514-kube-api-access-vp4k4\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tzrmz\" (UID: \"53045446-768f-47e4-9ef4-5eda1c261514\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tzrmz" Jul 28 01:22:15.102002 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:15.101982 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53045446-768f-47e4-9ef4-5eda1c261514-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tzrmz\" (UID: \"53045446-768f-47e4-9ef4-5eda1c261514\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tzrmz" Jul 28 01:22:15.102054 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:15.102002 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53045446-768f-47e4-9ef4-5eda1c261514-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tzrmz\" (UID: \"53045446-768f-47e4-9ef4-5eda1c261514\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tzrmz" Jul 28 01:22:15.110706 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:15.110678 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp4k4\" (UniqueName: \"kubernetes.io/projected/53045446-768f-47e4-9ef4-5eda1c261514-kube-api-access-vp4k4\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tzrmz\" (UID: \"53045446-768f-47e4-9ef4-5eda1c261514\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tzrmz" Jul 28 01:22:15.249630 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:15.249604 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tzrmz" Jul 28 01:22:15.335779 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:15.335748 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07dpr9"] Jul 28 01:22:15.339140 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:15.339108 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07dpr9" Jul 28 01:22:15.348227 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:15.347980 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07dpr9"] Jul 28 01:22:15.379638 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:15.379614 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tzrmz"] Jul 28 01:22:15.381345 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:22:15.381304 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53045446_768f_47e4_9ef4_5eda1c261514.slice/crio-8aafd834eabd4b06f40d3879085faad26a2d6d2b9c323bff4b519f4df0400e99 WatchSource:0}: Error finding container 8aafd834eabd4b06f40d3879085faad26a2d6d2b9c323bff4b519f4df0400e99: Status 404 returned error can't find the container with id 8aafd834eabd4b06f40d3879085faad26a2d6d2b9c323bff4b519f4df0400e99 Jul 28 01:22:15.403879 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:15.403857 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbfce4de-622d-48f4-af1e-eef982d353e2-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07dpr9\" (UID: \"fbfce4de-622d-48f4-af1e-eef982d353e2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07dpr9" Jul 28 01:22:15.403970 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:15.403898 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj6m6\" (UniqueName: \"kubernetes.io/projected/fbfce4de-622d-48f4-af1e-eef982d353e2-kube-api-access-xj6m6\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07dpr9\" (UID: \"fbfce4de-622d-48f4-af1e-eef982d353e2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07dpr9" Jul 28 01:22:15.403970 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:15.403937 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbfce4de-622d-48f4-af1e-eef982d353e2-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07dpr9\" (UID: \"fbfce4de-622d-48f4-af1e-eef982d353e2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07dpr9" Jul 28 01:22:15.505200 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:15.505120 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj6m6\" (UniqueName: \"kubernetes.io/projected/fbfce4de-622d-48f4-af1e-eef982d353e2-kube-api-access-xj6m6\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07dpr9\" (UID: \"fbfce4de-622d-48f4-af1e-eef982d353e2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07dpr9" Jul 28 01:22:15.505323 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:15.505195 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbfce4de-622d-48f4-af1e-eef982d353e2-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07dpr9\" (UID: \"fbfce4de-622d-48f4-af1e-eef982d353e2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07dpr9" Jul 28 01:22:15.505382 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:15.505333 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbfce4de-622d-48f4-af1e-eef982d353e2-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07dpr9\" (UID: \"fbfce4de-622d-48f4-af1e-eef982d353e2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07dpr9" Jul 28 01:22:15.505546 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:15.505528 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbfce4de-622d-48f4-af1e-eef982d353e2-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07dpr9\" (UID: \"fbfce4de-622d-48f4-af1e-eef982d353e2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07dpr9" Jul 28 01:22:15.505682 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:15.505647 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbfce4de-622d-48f4-af1e-eef982d353e2-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07dpr9\" (UID: \"fbfce4de-622d-48f4-af1e-eef982d353e2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07dpr9" Jul 28 01:22:15.514066 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:15.514018 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj6m6\" (UniqueName: \"kubernetes.io/projected/fbfce4de-622d-48f4-af1e-eef982d353e2-kube-api-access-xj6m6\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07dpr9\" (UID: \"fbfce4de-622d-48f4-af1e-eef982d353e2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07dpr9" Jul 28 01:22:15.650543 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:15.650518 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07dpr9" Jul 28 01:22:15.781632 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:15.781605 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07dpr9"] Jul 28 01:22:15.783961 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:22:15.783918 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbfce4de_622d_48f4_af1e_eef982d353e2.slice/crio-7bd68390640884b160fe4b357b4486331d4932e336cf70ac4c773f90418439ac WatchSource:0}: Error finding container 7bd68390640884b160fe4b357b4486331d4932e336cf70ac4c773f90418439ac: Status 404 returned error can't find the container with id 7bd68390640884b160fe4b357b4486331d4932e336cf70ac4c773f90418439ac Jul 28 01:22:15.789711 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:15.789685 2565 generic.go:356] "Generic (PLEG): container finished" podID="2ce1d280-4f01-4cae-8d9b-57d48994c94b" containerID="99bce5a5209d454fdec3ce3e4852be2028d1d9973ad8530381de9265d94d2771" exitCode=0 Jul 28 01:22:15.789795 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:15.789769 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g7xcs" event={"ID":"2ce1d280-4f01-4cae-8d9b-57d48994c94b","Type":"ContainerDied","Data":"99bce5a5209d454fdec3ce3e4852be2028d1d9973ad8530381de9265d94d2771"} Jul 28 01:22:15.791310 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:15.791284 2565 generic.go:356] "Generic (PLEG): container finished" podID="bb1f9dce-dee3-446c-a9bf-16764b20a710" containerID="8a08e2df132edda0e9dc664d9d0aac871ddeef268c5306a3c6506a95e0e82125" exitCode=0 Jul 28 01:22:15.791399 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:15.791358 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tptfg" event={"ID":"bb1f9dce-dee3-446c-a9bf-16764b20a710","Type":"ContainerDied","Data":"8a08e2df132edda0e9dc664d9d0aac871ddeef268c5306a3c6506a95e0e82125"} Jul 28 01:22:15.791449 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:15.791399 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tptfg" event={"ID":"bb1f9dce-dee3-446c-a9bf-16764b20a710","Type":"ContainerStarted","Data":"b476a33ae8803c7d3f233f69b29bc463aa35a3361c3d6f023f533c7f9672a411"} Jul 28 01:22:15.792974 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:15.792951 2565 generic.go:356] "Generic (PLEG): container finished" podID="53045446-768f-47e4-9ef4-5eda1c261514" containerID="50c32a18bcdef30291e3926bb6be4ac44b154a70f787c8c86ac1a496ca228de5" exitCode=0 Jul 28 01:22:15.793075 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:15.793004 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tzrmz" event={"ID":"53045446-768f-47e4-9ef4-5eda1c261514","Type":"ContainerDied","Data":"50c32a18bcdef30291e3926bb6be4ac44b154a70f787c8c86ac1a496ca228de5"} Jul 28 01:22:15.793075 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:15.793066 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tzrmz" event={"ID":"53045446-768f-47e4-9ef4-5eda1c261514","Type":"ContainerStarted","Data":"8aafd834eabd4b06f40d3879085faad26a2d6d2b9c323bff4b519f4df0400e99"} Jul 28 01:22:16.799352 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:16.799318 2565 generic.go:356] "Generic (PLEG): container finished" podID="2ce1d280-4f01-4cae-8d9b-57d48994c94b" containerID="71b25a7fcfc03604cd7b732d7746495f5ac2880f6eb90df636550f3db9fb087b" exitCode=0 Jul 28 01:22:16.799738 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:16.799406 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g7xcs" event={"ID":"2ce1d280-4f01-4cae-8d9b-57d48994c94b","Type":"ContainerDied","Data":"71b25a7fcfc03604cd7b732d7746495f5ac2880f6eb90df636550f3db9fb087b"} Jul 28 01:22:16.801142 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:16.801120 2565 generic.go:356] "Generic (PLEG): container finished" podID="bb1f9dce-dee3-446c-a9bf-16764b20a710" containerID="35570e9ea8ccfd3b397859d125a9f40488d09a16aea51d956e1db99bba9cf92d" exitCode=0 Jul 28 01:22:16.801248 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:16.801208 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tptfg" event={"ID":"bb1f9dce-dee3-446c-a9bf-16764b20a710","Type":"ContainerDied","Data":"35570e9ea8ccfd3b397859d125a9f40488d09a16aea51d956e1db99bba9cf92d"} Jul 28 01:22:16.802760 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:16.802741 2565 generic.go:356] "Generic (PLEG): container finished" podID="fbfce4de-622d-48f4-af1e-eef982d353e2" containerID="ad1788c7c53df6851f1791cd73d49c9c6a28e5a9f1ce752f184fc85f7aae6c1b" exitCode=0 Jul 28 01:22:16.802846 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:16.802813 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07dpr9" event={"ID":"fbfce4de-622d-48f4-af1e-eef982d353e2","Type":"ContainerDied","Data":"ad1788c7c53df6851f1791cd73d49c9c6a28e5a9f1ce752f184fc85f7aae6c1b"} Jul 28 01:22:16.802902 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:16.802849 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07dpr9" event={"ID":"fbfce4de-622d-48f4-af1e-eef982d353e2","Type":"ContainerStarted","Data":"7bd68390640884b160fe4b357b4486331d4932e336cf70ac4c773f90418439ac"} Jul 28 01:22:16.804698 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:16.804669 2565 generic.go:356] "Generic (PLEG): container finished" podID="53045446-768f-47e4-9ef4-5eda1c261514" containerID="a691a2054c15e9852f4878749b2100e3ab61607e7a6300cd2ec5e80ddb1e7c52" exitCode=0 Jul 28 01:22:16.804796 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:16.804706 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tzrmz" event={"ID":"53045446-768f-47e4-9ef4-5eda1c261514","Type":"ContainerDied","Data":"a691a2054c15e9852f4878749b2100e3ab61607e7a6300cd2ec5e80ddb1e7c52"} Jul 28 01:22:17.811398 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:17.811369 2565 generic.go:356] "Generic (PLEG): container finished" podID="53045446-768f-47e4-9ef4-5eda1c261514" containerID="69c130ecb8abf071ff4fcc850853ddeec7adb33098cd60e78b4364ef5663a61d" exitCode=0 Jul 28 01:22:17.811761 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:17.811421 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tzrmz" event={"ID":"53045446-768f-47e4-9ef4-5eda1c261514","Type":"ContainerDied","Data":"69c130ecb8abf071ff4fcc850853ddeec7adb33098cd60e78b4364ef5663a61d"} Jul 28 01:22:17.813697 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:17.813671 2565 generic.go:356] "Generic (PLEG): container finished" podID="bb1f9dce-dee3-446c-a9bf-16764b20a710" containerID="f551b598892db42db3560e826fe3912dd053c8e77440eb39adb7bfb91cc1cb20" exitCode=0 Jul 28 01:22:17.813796 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:17.813762 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tptfg" event={"ID":"bb1f9dce-dee3-446c-a9bf-16764b20a710","Type":"ContainerDied","Data":"f551b598892db42db3560e826fe3912dd053c8e77440eb39adb7bfb91cc1cb20"} Jul 28 01:22:17.972601 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:17.972579 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g7xcs" Jul 28 01:22:18.023146 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:18.023117 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ce1d280-4f01-4cae-8d9b-57d48994c94b-util\") pod \"2ce1d280-4f01-4cae-8d9b-57d48994c94b\" (UID: \"2ce1d280-4f01-4cae-8d9b-57d48994c94b\") " Jul 28 01:22:18.023294 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:18.023160 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smc7l\" (UniqueName: \"kubernetes.io/projected/2ce1d280-4f01-4cae-8d9b-57d48994c94b-kube-api-access-smc7l\") pod \"2ce1d280-4f01-4cae-8d9b-57d48994c94b\" (UID: \"2ce1d280-4f01-4cae-8d9b-57d48994c94b\") " Jul 28 01:22:18.023294 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:18.023179 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ce1d280-4f01-4cae-8d9b-57d48994c94b-bundle\") pod \"2ce1d280-4f01-4cae-8d9b-57d48994c94b\" (UID: \"2ce1d280-4f01-4cae-8d9b-57d48994c94b\") " Jul 28 01:22:18.023670 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:18.023641 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce1d280-4f01-4cae-8d9b-57d48994c94b-bundle" (OuterVolumeSpecName: "bundle") pod "2ce1d280-4f01-4cae-8d9b-57d48994c94b" (UID: "2ce1d280-4f01-4cae-8d9b-57d48994c94b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:22:18.025277 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:18.025251 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ce1d280-4f01-4cae-8d9b-57d48994c94b-kube-api-access-smc7l" (OuterVolumeSpecName: "kube-api-access-smc7l") pod "2ce1d280-4f01-4cae-8d9b-57d48994c94b" (UID: "2ce1d280-4f01-4cae-8d9b-57d48994c94b"). InnerVolumeSpecName "kube-api-access-smc7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:22:18.028746 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:18.028728 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce1d280-4f01-4cae-8d9b-57d48994c94b-util" (OuterVolumeSpecName: "util") pod "2ce1d280-4f01-4cae-8d9b-57d48994c94b" (UID: "2ce1d280-4f01-4cae-8d9b-57d48994c94b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:22:18.124412 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:18.124313 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ce1d280-4f01-4cae-8d9b-57d48994c94b-util\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:22:18.124412 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:18.124339 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smc7l\" (UniqueName: \"kubernetes.io/projected/2ce1d280-4f01-4cae-8d9b-57d48994c94b-kube-api-access-smc7l\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:22:18.124412 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:18.124351 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ce1d280-4f01-4cae-8d9b-57d48994c94b-bundle\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:22:18.823122 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:18.823088 2565 generic.go:356] "Generic (PLEG): container finished" podID="fbfce4de-622d-48f4-af1e-eef982d353e2" containerID="e81263546d4319b1431a0252107baf7d82bd6ddccd55e5b4403144fcc9a6957e" exitCode=0 Jul 28 01:22:18.823578 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:18.823171 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07dpr9" event={"ID":"fbfce4de-622d-48f4-af1e-eef982d353e2","Type":"ContainerDied","Data":"e81263546d4319b1431a0252107baf7d82bd6ddccd55e5b4403144fcc9a6957e"} Jul 28 01:22:18.825094 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:18.825077 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g7xcs" Jul 28 01:22:18.825183 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:18.825106 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g7xcs" event={"ID":"2ce1d280-4f01-4cae-8d9b-57d48994c94b","Type":"ContainerDied","Data":"da73b2585fbaf5c1c236def5789d414d7ddd13d78004cdcb3e96c401574527e9"} Jul 28 01:22:18.825183 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:18.825147 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da73b2585fbaf5c1c236def5789d414d7ddd13d78004cdcb3e96c401574527e9" Jul 28 01:22:19.045172 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:19.045149 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tzrmz" Jul 28 01:22:19.047376 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:19.047356 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tptfg" Jul 28 01:22:19.133384 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:19.133301 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53045446-768f-47e4-9ef4-5eda1c261514-bundle\") pod \"53045446-768f-47e4-9ef4-5eda1c261514\" (UID: \"53045446-768f-47e4-9ef4-5eda1c261514\") " Jul 28 01:22:19.133384 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:19.133348 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53045446-768f-47e4-9ef4-5eda1c261514-util\") pod \"53045446-768f-47e4-9ef4-5eda1c261514\" (UID: \"53045446-768f-47e4-9ef4-5eda1c261514\") " Jul 28 01:22:19.133384 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:19.133371 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb1f9dce-dee3-446c-a9bf-16764b20a710-bundle\") pod \"bb1f9dce-dee3-446c-a9bf-16764b20a710\" (UID: \"bb1f9dce-dee3-446c-a9bf-16764b20a710\") " Jul 28 01:22:19.133619 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:19.133401 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vp4k4\" (UniqueName: \"kubernetes.io/projected/53045446-768f-47e4-9ef4-5eda1c261514-kube-api-access-vp4k4\") pod \"53045446-768f-47e4-9ef4-5eda1c261514\" (UID: \"53045446-768f-47e4-9ef4-5eda1c261514\") " Jul 28 01:22:19.133619 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:19.133424 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb1f9dce-dee3-446c-a9bf-16764b20a710-util\") pod \"bb1f9dce-dee3-446c-a9bf-16764b20a710\" (UID: \"bb1f9dce-dee3-446c-a9bf-16764b20a710\") " Jul 28 01:22:19.133619 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:19.133444 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gq5n2\" (UniqueName: \"kubernetes.io/projected/bb1f9dce-dee3-446c-a9bf-16764b20a710-kube-api-access-gq5n2\") pod \"bb1f9dce-dee3-446c-a9bf-16764b20a710\" (UID: \"bb1f9dce-dee3-446c-a9bf-16764b20a710\") " Jul 28 01:22:19.134109 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:19.134024 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53045446-768f-47e4-9ef4-5eda1c261514-bundle" (OuterVolumeSpecName: "bundle") pod "53045446-768f-47e4-9ef4-5eda1c261514" (UID: "53045446-768f-47e4-9ef4-5eda1c261514"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:22:19.134236 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:19.134122 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1f9dce-dee3-446c-a9bf-16764b20a710-bundle" (OuterVolumeSpecName: "bundle") pod "bb1f9dce-dee3-446c-a9bf-16764b20a710" (UID: "bb1f9dce-dee3-446c-a9bf-16764b20a710"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:22:19.135743 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:19.135724 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53045446-768f-47e4-9ef4-5eda1c261514-kube-api-access-vp4k4" (OuterVolumeSpecName: "kube-api-access-vp4k4") pod "53045446-768f-47e4-9ef4-5eda1c261514" (UID: "53045446-768f-47e4-9ef4-5eda1c261514"). InnerVolumeSpecName "kube-api-access-vp4k4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:22:19.135871 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:19.135852 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb1f9dce-dee3-446c-a9bf-16764b20a710-kube-api-access-gq5n2" (OuterVolumeSpecName: "kube-api-access-gq5n2") pod "bb1f9dce-dee3-446c-a9bf-16764b20a710" (UID: "bb1f9dce-dee3-446c-a9bf-16764b20a710"). InnerVolumeSpecName "kube-api-access-gq5n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:22:19.138495 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:19.138473 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1f9dce-dee3-446c-a9bf-16764b20a710-util" (OuterVolumeSpecName: "util") pod "bb1f9dce-dee3-446c-a9bf-16764b20a710" (UID: "bb1f9dce-dee3-446c-a9bf-16764b20a710"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:22:19.139324 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:19.139305 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53045446-768f-47e4-9ef4-5eda1c261514-util" (OuterVolumeSpecName: "util") pod "53045446-768f-47e4-9ef4-5eda1c261514" (UID: "53045446-768f-47e4-9ef4-5eda1c261514"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:22:19.234595 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:19.234561 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53045446-768f-47e4-9ef4-5eda1c261514-bundle\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:22:19.234595 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:19.234588 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53045446-768f-47e4-9ef4-5eda1c261514-util\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:22:19.234595 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:19.234598 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb1f9dce-dee3-446c-a9bf-16764b20a710-bundle\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:22:19.234887 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:19.234607 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp4k4\" (UniqueName: \"kubernetes.io/projected/53045446-768f-47e4-9ef4-5eda1c261514-kube-api-access-vp4k4\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:22:19.234887 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:19.234616 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb1f9dce-dee3-446c-a9bf-16764b20a710-util\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:22:19.234887 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:19.234624 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq5n2\" (UniqueName: \"kubernetes.io/projected/bb1f9dce-dee3-446c-a9bf-16764b20a710-kube-api-access-gq5n2\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:22:19.831281 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:19.831253 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tptfg" Jul 28 01:22:19.831740 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:19.831277 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tptfg" event={"ID":"bb1f9dce-dee3-446c-a9bf-16764b20a710","Type":"ContainerDied","Data":"b476a33ae8803c7d3f233f69b29bc463aa35a3361c3d6f023f533c7f9672a411"} Jul 28 01:22:19.831740 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:19.831329 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b476a33ae8803c7d3f233f69b29bc463aa35a3361c3d6f023f533c7f9672a411" Jul 28 01:22:19.833393 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:19.833367 2565 generic.go:356] "Generic (PLEG): container finished" podID="fbfce4de-622d-48f4-af1e-eef982d353e2" containerID="675a85f7294e69076f27b39c1387a04f90d09934558cfb3e5bdc21b7e3caba44" exitCode=0 Jul 28 01:22:19.833543 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:19.833452 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07dpr9" event={"ID":"fbfce4de-622d-48f4-af1e-eef982d353e2","Type":"ContainerDied","Data":"675a85f7294e69076f27b39c1387a04f90d09934558cfb3e5bdc21b7e3caba44"} Jul 28 01:22:19.835203 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:19.835180 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tzrmz" event={"ID":"53045446-768f-47e4-9ef4-5eda1c261514","Type":"ContainerDied","Data":"8aafd834eabd4b06f40d3879085faad26a2d6d2b9c323bff4b519f4df0400e99"} Jul 28 01:22:19.835339 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:19.835212 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8aafd834eabd4b06f40d3879085faad26a2d6d2b9c323bff4b519f4df0400e99" Jul 28 01:22:19.835339 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:19.835215 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tzrmz" Jul 28 01:22:20.967831 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:20.967809 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07dpr9" Jul 28 01:22:21.051481 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:21.051454 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbfce4de-622d-48f4-af1e-eef982d353e2-util\") pod \"fbfce4de-622d-48f4-af1e-eef982d353e2\" (UID: \"fbfce4de-622d-48f4-af1e-eef982d353e2\") " Jul 28 01:22:21.051618 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:21.051514 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xj6m6\" (UniqueName: \"kubernetes.io/projected/fbfce4de-622d-48f4-af1e-eef982d353e2-kube-api-access-xj6m6\") pod \"fbfce4de-622d-48f4-af1e-eef982d353e2\" (UID: \"fbfce4de-622d-48f4-af1e-eef982d353e2\") " Jul 28 01:22:21.051618 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:21.051564 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbfce4de-622d-48f4-af1e-eef982d353e2-bundle\") pod \"fbfce4de-622d-48f4-af1e-eef982d353e2\" (UID: \"fbfce4de-622d-48f4-af1e-eef982d353e2\") " Jul 28 01:22:21.052180 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:21.052120 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbfce4de-622d-48f4-af1e-eef982d353e2-bundle" (OuterVolumeSpecName: "bundle") pod "fbfce4de-622d-48f4-af1e-eef982d353e2" (UID: "fbfce4de-622d-48f4-af1e-eef982d353e2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:22:21.053578 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:21.053556 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbfce4de-622d-48f4-af1e-eef982d353e2-kube-api-access-xj6m6" (OuterVolumeSpecName: "kube-api-access-xj6m6") pod "fbfce4de-622d-48f4-af1e-eef982d353e2" (UID: "fbfce4de-622d-48f4-af1e-eef982d353e2"). InnerVolumeSpecName "kube-api-access-xj6m6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:22:21.058316 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:21.058289 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbfce4de-622d-48f4-af1e-eef982d353e2-util" (OuterVolumeSpecName: "util") pod "fbfce4de-622d-48f4-af1e-eef982d353e2" (UID: "fbfce4de-622d-48f4-af1e-eef982d353e2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:22:21.152521 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:21.152495 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbfce4de-622d-48f4-af1e-eef982d353e2-util\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:22:21.152521 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:21.152519 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj6m6\" (UniqueName: \"kubernetes.io/projected/fbfce4de-622d-48f4-af1e-eef982d353e2-kube-api-access-xj6m6\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:22:21.152688 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:21.152530 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbfce4de-622d-48f4-af1e-eef982d353e2-bundle\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:22:21.851763 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:21.851736 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07dpr9" Jul 28 01:22:21.851763 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:21.851752 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07dpr9" event={"ID":"fbfce4de-622d-48f4-af1e-eef982d353e2","Type":"ContainerDied","Data":"7bd68390640884b160fe4b357b4486331d4932e336cf70ac4c773f90418439ac"} Jul 28 01:22:21.852010 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:21.851791 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bd68390640884b160fe4b357b4486331d4932e336cf70ac4c773f90418439ac" Jul 28 01:22:40.389576 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:40.389542 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58df9db6f4-kxmqm"] Jul 28 01:22:58.011543 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.011511 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p47xv"] Jul 28 01:22:58.012005 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.011807 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53045446-768f-47e4-9ef4-5eda1c261514" containerName="util" Jul 28 01:22:58.012005 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.011817 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="53045446-768f-47e4-9ef4-5eda1c261514" containerName="util" Jul 28 01:22:58.012005 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.011826 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ce1d280-4f01-4cae-8d9b-57d48994c94b" containerName="pull" Jul 28 01:22:58.012005 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.011831 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ce1d280-4f01-4cae-8d9b-57d48994c94b" containerName="pull" Jul 28 01:22:58.012005 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.011838 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ce1d280-4f01-4cae-8d9b-57d48994c94b" containerName="extract" Jul 28 01:22:58.012005 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.011844 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ce1d280-4f01-4cae-8d9b-57d48994c94b" containerName="extract" Jul 28 01:22:58.012005 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.011851 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53045446-768f-47e4-9ef4-5eda1c261514" containerName="pull" Jul 28 01:22:58.012005 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.011856 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="53045446-768f-47e4-9ef4-5eda1c261514" containerName="pull" Jul 28 01:22:58.012005 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.011867 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb1f9dce-dee3-446c-a9bf-16764b20a710" containerName="pull" Jul 28 01:22:58.012005 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.011872 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb1f9dce-dee3-446c-a9bf-16764b20a710" containerName="pull" Jul 28 01:22:58.012005 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.011879 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbfce4de-622d-48f4-af1e-eef982d353e2" containerName="extract" Jul 28 01:22:58.012005 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.011884 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbfce4de-622d-48f4-af1e-eef982d353e2" containerName="extract" Jul 28 01:22:58.012005 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.011892 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ce1d280-4f01-4cae-8d9b-57d48994c94b" containerName="util" Jul 28 01:22:58.012005 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.011897 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ce1d280-4f01-4cae-8d9b-57d48994c94b" containerName="util" Jul 28 01:22:58.012005 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.011903 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb1f9dce-dee3-446c-a9bf-16764b20a710" containerName="extract" Jul 28 01:22:58.012005 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.011908 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb1f9dce-dee3-446c-a9bf-16764b20a710" containerName="extract" Jul 28 01:22:58.012005 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.011915 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb1f9dce-dee3-446c-a9bf-16764b20a710" containerName="util" Jul 28 01:22:58.012005 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.011921 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb1f9dce-dee3-446c-a9bf-16764b20a710" containerName="util" Jul 28 01:22:58.012005 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.011926 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbfce4de-622d-48f4-af1e-eef982d353e2" containerName="util" Jul 28 01:22:58.012005 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.011931 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbfce4de-622d-48f4-af1e-eef982d353e2" containerName="util" Jul 28 01:22:58.012005 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.011936 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53045446-768f-47e4-9ef4-5eda1c261514" containerName="extract" Jul 28 01:22:58.012005 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.011940 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="53045446-768f-47e4-9ef4-5eda1c261514" containerName="extract" Jul 28 01:22:58.012005 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.011947 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbfce4de-622d-48f4-af1e-eef982d353e2" containerName="pull" Jul 28 01:22:58.012005 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.011952 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbfce4de-622d-48f4-af1e-eef982d353e2" containerName="pull" Jul 28 01:22:58.012005 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.011995 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb1f9dce-dee3-446c-a9bf-16764b20a710" containerName="extract" Jul 28 01:22:58.012005 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.012002 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="53045446-768f-47e4-9ef4-5eda1c261514" containerName="extract" Jul 28 01:22:58.012005 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.012008 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbfce4de-622d-48f4-af1e-eef982d353e2" containerName="extract" Jul 28 01:22:58.012005 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.012015 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ce1d280-4f01-4cae-8d9b-57d48994c94b" containerName="extract" Jul 28 01:22:58.015326 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.015298 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p47xv" Jul 28 01:22:58.018564 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.018537 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 28 01:22:58.018712 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.018685 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-bxzdn\"" Jul 28 01:22:58.019795 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.019776 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 28 01:22:58.025650 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.025626 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p47xv"] Jul 28 01:22:58.164384 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.164342 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wnp5\" (UniqueName: \"kubernetes.io/projected/e779e369-b9ef-4a41-8596-ee74986e44f1-kube-api-access-4wnp5\") pod \"limitador-operator-controller-manager-859d7b78bd-p47xv\" (UID: \"e779e369-b9ef-4a41-8596-ee74986e44f1\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p47xv" Jul 28 01:22:58.265574 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.265484 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wnp5\" (UniqueName: \"kubernetes.io/projected/e779e369-b9ef-4a41-8596-ee74986e44f1-kube-api-access-4wnp5\") pod \"limitador-operator-controller-manager-859d7b78bd-p47xv\" (UID: \"e779e369-b9ef-4a41-8596-ee74986e44f1\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p47xv" Jul 28 01:22:58.275791 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.275752 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wnp5\" (UniqueName: \"kubernetes.io/projected/e779e369-b9ef-4a41-8596-ee74986e44f1-kube-api-access-4wnp5\") pod \"limitador-operator-controller-manager-859d7b78bd-p47xv\" (UID: \"e779e369-b9ef-4a41-8596-ee74986e44f1\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p47xv" Jul 28 01:22:58.328299 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.328265 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p47xv" Jul 28 01:22:58.462428 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:58.462404 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p47xv"] Jul 28 01:22:58.464571 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:22:58.464541 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode779e369_b9ef_4a41_8596_ee74986e44f1.slice/crio-e83bf5f065b70239775c94e18843341ddb21216681ffd184dae4940e3061eb8c WatchSource:0}: Error finding container e83bf5f065b70239775c94e18843341ddb21216681ffd184dae4940e3061eb8c: Status 404 returned error can't find the container with id e83bf5f065b70239775c94e18843341ddb21216681ffd184dae4940e3061eb8c Jul 28 01:22:59.026374 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:22:59.026338 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p47xv" event={"ID":"e779e369-b9ef-4a41-8596-ee74986e44f1","Type":"ContainerStarted","Data":"e83bf5f065b70239775c94e18843341ddb21216681ffd184dae4940e3061eb8c"} Jul 28 01:23:00.032742 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:00.032706 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p47xv" event={"ID":"e779e369-b9ef-4a41-8596-ee74986e44f1","Type":"ContainerStarted","Data":"4134a056c0c61c80856f875388128f009725ecb4d323a51fa38b66846e9b931e"} Jul 28 01:23:00.033166 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:00.032865 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p47xv" Jul 28 01:23:00.054766 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:00.054714 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p47xv" podStartSLOduration=1.685365073 podStartE2EDuration="3.054698676s" totalImagesPullingTime="1.369333603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:23:00.053134919 +0000 UTC m=+437.487533237" watchObservedRunningTime="2026-07-28 01:23:00.054698676 +0000 UTC m=+437.489096991" Jul 28 01:23:05.410187 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:05.410104 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-58df9db6f4-kxmqm" podUID="420dc905-c3d6-4b00-b0dd-3e76542c1801" containerName="console" containerID="cri-o://6a04716f99de47c22006ab21ce2b67d149f1c037b0e42416ba880e1327477e2b" gracePeriod=15 Jul 28 01:23:05.654947 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:05.654922 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58df9db6f4-kxmqm" Jul 28 01:23:05.731210 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:05.731175 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/420dc905-c3d6-4b00-b0dd-3e76542c1801-console-oauth-config\") pod \"420dc905-c3d6-4b00-b0dd-3e76542c1801\" (UID: \"420dc905-c3d6-4b00-b0dd-3e76542c1801\") " Jul 28 01:23:05.731405 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:05.731235 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/420dc905-c3d6-4b00-b0dd-3e76542c1801-service-ca\") pod \"420dc905-c3d6-4b00-b0dd-3e76542c1801\" (UID: \"420dc905-c3d6-4b00-b0dd-3e76542c1801\") " Jul 28 01:23:05.731405 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:05.731260 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/420dc905-c3d6-4b00-b0dd-3e76542c1801-console-config\") pod \"420dc905-c3d6-4b00-b0dd-3e76542c1801\" (UID: \"420dc905-c3d6-4b00-b0dd-3e76542c1801\") " Jul 28 01:23:05.731405 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:05.731286 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98jvw\" (UniqueName: \"kubernetes.io/projected/420dc905-c3d6-4b00-b0dd-3e76542c1801-kube-api-access-98jvw\") pod \"420dc905-c3d6-4b00-b0dd-3e76542c1801\" (UID: \"420dc905-c3d6-4b00-b0dd-3e76542c1801\") " Jul 28 01:23:05.731405 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:05.731323 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/420dc905-c3d6-4b00-b0dd-3e76542c1801-console-serving-cert\") pod \"420dc905-c3d6-4b00-b0dd-3e76542c1801\" (UID: \"420dc905-c3d6-4b00-b0dd-3e76542c1801\") " Jul 28 01:23:05.731619 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:05.731470 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/420dc905-c3d6-4b00-b0dd-3e76542c1801-oauth-serving-cert\") pod \"420dc905-c3d6-4b00-b0dd-3e76542c1801\" (UID: \"420dc905-c3d6-4b00-b0dd-3e76542c1801\") " Jul 28 01:23:05.731619 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:05.731516 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/420dc905-c3d6-4b00-b0dd-3e76542c1801-trusted-ca-bundle\") pod \"420dc905-c3d6-4b00-b0dd-3e76542c1801\" (UID: \"420dc905-c3d6-4b00-b0dd-3e76542c1801\") " Jul 28 01:23:05.731739 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:05.731702 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/420dc905-c3d6-4b00-b0dd-3e76542c1801-service-ca" (OuterVolumeSpecName: "service-ca") pod "420dc905-c3d6-4b00-b0dd-3e76542c1801" (UID: "420dc905-c3d6-4b00-b0dd-3e76542c1801"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:23:05.731796 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:05.731732 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/420dc905-c3d6-4b00-b0dd-3e76542c1801-console-config" (OuterVolumeSpecName: "console-config") pod "420dc905-c3d6-4b00-b0dd-3e76542c1801" (UID: "420dc905-c3d6-4b00-b0dd-3e76542c1801"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:23:05.731847 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:05.731812 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/420dc905-c3d6-4b00-b0dd-3e76542c1801-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "420dc905-c3d6-4b00-b0dd-3e76542c1801" (UID: "420dc905-c3d6-4b00-b0dd-3e76542c1801"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:23:05.731847 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:05.731837 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/420dc905-c3d6-4b00-b0dd-3e76542c1801-service-ca\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:23:05.731944 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:05.731855 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/420dc905-c3d6-4b00-b0dd-3e76542c1801-console-config\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:23:05.732047 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:05.731989 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/420dc905-c3d6-4b00-b0dd-3e76542c1801-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "420dc905-c3d6-4b00-b0dd-3e76542c1801" (UID: "420dc905-c3d6-4b00-b0dd-3e76542c1801"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:23:05.733557 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:05.733533 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/420dc905-c3d6-4b00-b0dd-3e76542c1801-kube-api-access-98jvw" (OuterVolumeSpecName: "kube-api-access-98jvw") pod "420dc905-c3d6-4b00-b0dd-3e76542c1801" (UID: "420dc905-c3d6-4b00-b0dd-3e76542c1801"). InnerVolumeSpecName "kube-api-access-98jvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:23:05.733888 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:05.733860 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/420dc905-c3d6-4b00-b0dd-3e76542c1801-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "420dc905-c3d6-4b00-b0dd-3e76542c1801" (UID: "420dc905-c3d6-4b00-b0dd-3e76542c1801"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:23:05.733975 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:05.733927 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/420dc905-c3d6-4b00-b0dd-3e76542c1801-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "420dc905-c3d6-4b00-b0dd-3e76542c1801" (UID: "420dc905-c3d6-4b00-b0dd-3e76542c1801"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:23:05.832290 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:05.832254 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/420dc905-c3d6-4b00-b0dd-3e76542c1801-console-serving-cert\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:23:05.832290 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:05.832283 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/420dc905-c3d6-4b00-b0dd-3e76542c1801-oauth-serving-cert\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:23:05.832290 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:05.832292 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/420dc905-c3d6-4b00-b0dd-3e76542c1801-trusted-ca-bundle\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:23:05.832290 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:05.832299 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/420dc905-c3d6-4b00-b0dd-3e76542c1801-console-oauth-config\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:23:05.832534 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:05.832310 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98jvw\" (UniqueName: \"kubernetes.io/projected/420dc905-c3d6-4b00-b0dd-3e76542c1801-kube-api-access-98jvw\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:23:06.062628 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:06.062540 2565 generic.go:356] "Generic (PLEG): container finished" podID="420dc905-c3d6-4b00-b0dd-3e76542c1801" containerID="6a04716f99de47c22006ab21ce2b67d149f1c037b0e42416ba880e1327477e2b" exitCode=0 Jul 28 01:23:06.062628 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:06.062586 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58df9db6f4-kxmqm" Jul 28 01:23:06.062807 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:06.062629 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58df9db6f4-kxmqm" event={"ID":"420dc905-c3d6-4b00-b0dd-3e76542c1801","Type":"ContainerDied","Data":"6a04716f99de47c22006ab21ce2b67d149f1c037b0e42416ba880e1327477e2b"} Jul 28 01:23:06.062807 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:06.062695 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58df9db6f4-kxmqm" event={"ID":"420dc905-c3d6-4b00-b0dd-3e76542c1801","Type":"ContainerDied","Data":"e0aa16ae9c3d5f81f4757b88173aef6d6e95171203923466c1cc16f63f1b12a7"} Jul 28 01:23:06.062807 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:06.062716 2565 scope.go:117] "RemoveContainer" containerID="6a04716f99de47c22006ab21ce2b67d149f1c037b0e42416ba880e1327477e2b" Jul 28 01:23:06.071411 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:06.071394 2565 scope.go:117] "RemoveContainer" containerID="6a04716f99de47c22006ab21ce2b67d149f1c037b0e42416ba880e1327477e2b" Jul 28 01:23:06.071698 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:23:06.071681 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a04716f99de47c22006ab21ce2b67d149f1c037b0e42416ba880e1327477e2b\": container with ID starting with 6a04716f99de47c22006ab21ce2b67d149f1c037b0e42416ba880e1327477e2b not found: ID does not exist" containerID="6a04716f99de47c22006ab21ce2b67d149f1c037b0e42416ba880e1327477e2b" Jul 28 01:23:06.071745 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:06.071706 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a04716f99de47c22006ab21ce2b67d149f1c037b0e42416ba880e1327477e2b"} err="failed to get container status \"6a04716f99de47c22006ab21ce2b67d149f1c037b0e42416ba880e1327477e2b\": rpc error: code = NotFound desc = could not find container \"6a04716f99de47c22006ab21ce2b67d149f1c037b0e42416ba880e1327477e2b\": container with ID starting with 6a04716f99de47c22006ab21ce2b67d149f1c037b0e42416ba880e1327477e2b not found: ID does not exist" Jul 28 01:23:06.084711 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:06.084689 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58df9db6f4-kxmqm"] Jul 28 01:23:06.087936 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:06.087904 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58df9db6f4-kxmqm"] Jul 28 01:23:07.100259 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:07.100224 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="420dc905-c3d6-4b00-b0dd-3e76542c1801" path="/var/lib/kubelet/pods/420dc905-c3d6-4b00-b0dd-3e76542c1801/volumes" Jul 28 01:23:11.040476 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:23:11.040448 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p47xv" Jul 28 01:25:50.978679 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:25:50.978642 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk"] Jul 28 01:25:50.981005 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:25:50.978928 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="420dc905-c3d6-4b00-b0dd-3e76542c1801" containerName="console" Jul 28 01:25:50.981005 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:25:50.978939 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="420dc905-c3d6-4b00-b0dd-3e76542c1801" containerName="console" Jul 28 01:25:50.981005 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:25:50.979013 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="420dc905-c3d6-4b00-b0dd-3e76542c1801" containerName="console" Jul 28 01:25:50.981905 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:25:50.981889 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk" Jul 28 01:25:50.985557 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:25:50.985534 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 28 01:25:50.985681 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:25:50.985564 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-rtwcb\"" Jul 28 01:25:50.985681 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:25:50.985534 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 28 01:25:50.985681 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:25:50.985586 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 28 01:25:50.993027 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:25:50.993003 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk"] Jul 28 01:25:51.060623 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:25:51.060586 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dbcb09e-a463-427d-8e2a-0023c811ec1d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk\" (UID: \"7dbcb09e-a463-427d-8e2a-0023c811ec1d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk" Jul 28 01:25:51.060623 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:25:51.060629 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dbcb09e-a463-427d-8e2a-0023c811ec1d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk\" (UID: \"7dbcb09e-a463-427d-8e2a-0023c811ec1d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk" Jul 28 01:25:51.060854 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:25:51.060646 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dbcb09e-a463-427d-8e2a-0023c811ec1d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk\" (UID: \"7dbcb09e-a463-427d-8e2a-0023c811ec1d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk" Jul 28 01:25:51.060854 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:25:51.060707 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dbcb09e-a463-427d-8e2a-0023c811ec1d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk\" (UID: \"7dbcb09e-a463-427d-8e2a-0023c811ec1d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk" Jul 28 01:25:51.060854 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:25:51.060729 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dbcb09e-a463-427d-8e2a-0023c811ec1d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk\" (UID: \"7dbcb09e-a463-427d-8e2a-0023c811ec1d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk" Jul 28 01:25:51.060854 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:25:51.060750 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnxtp\" (UniqueName: \"kubernetes.io/projected/7dbcb09e-a463-427d-8e2a-0023c811ec1d-kube-api-access-bnxtp\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk\" (UID: \"7dbcb09e-a463-427d-8e2a-0023c811ec1d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk" Jul 28 01:25:51.162135 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:25:51.162100 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dbcb09e-a463-427d-8e2a-0023c811ec1d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk\" (UID: \"7dbcb09e-a463-427d-8e2a-0023c811ec1d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk" Jul 28 01:25:51.162302 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:25:51.162144 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dbcb09e-a463-427d-8e2a-0023c811ec1d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk\" (UID: \"7dbcb09e-a463-427d-8e2a-0023c811ec1d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk" Jul 28 01:25:51.162302 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:25:51.162170 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dbcb09e-a463-427d-8e2a-0023c811ec1d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk\" (UID: \"7dbcb09e-a463-427d-8e2a-0023c811ec1d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk" Jul 28 01:25:51.162302 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:25:51.162219 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dbcb09e-a463-427d-8e2a-0023c811ec1d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk\" (UID: \"7dbcb09e-a463-427d-8e2a-0023c811ec1d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk" Jul 28 01:25:51.162302 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:25:51.162252 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dbcb09e-a463-427d-8e2a-0023c811ec1d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk\" (UID: \"7dbcb09e-a463-427d-8e2a-0023c811ec1d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk" Jul 28 01:25:51.162302 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:25:51.162272 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnxtp\" (UniqueName: \"kubernetes.io/projected/7dbcb09e-a463-427d-8e2a-0023c811ec1d-kube-api-access-bnxtp\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk\" (UID: \"7dbcb09e-a463-427d-8e2a-0023c811ec1d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk" Jul 28 01:25:51.162594 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:25:51.162569 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dbcb09e-a463-427d-8e2a-0023c811ec1d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk\" (UID: \"7dbcb09e-a463-427d-8e2a-0023c811ec1d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk" Jul 28 01:25:51.162640 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:25:51.162592 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dbcb09e-a463-427d-8e2a-0023c811ec1d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk\" (UID: \"7dbcb09e-a463-427d-8e2a-0023c811ec1d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk" Jul 28 01:25:51.162678 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:25:51.162665 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dbcb09e-a463-427d-8e2a-0023c811ec1d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk\" (UID: \"7dbcb09e-a463-427d-8e2a-0023c811ec1d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk" Jul 28 01:25:51.164555 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:25:51.164538 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dbcb09e-a463-427d-8e2a-0023c811ec1d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk\" (UID: \"7dbcb09e-a463-427d-8e2a-0023c811ec1d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk" Jul 28 01:25:51.164686 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:25:51.164670 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dbcb09e-a463-427d-8e2a-0023c811ec1d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk\" (UID: \"7dbcb09e-a463-427d-8e2a-0023c811ec1d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk" Jul 28 01:25:51.170121 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:25:51.170099 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnxtp\" (UniqueName: \"kubernetes.io/projected/7dbcb09e-a463-427d-8e2a-0023c811ec1d-kube-api-access-bnxtp\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk\" (UID: \"7dbcb09e-a463-427d-8e2a-0023c811ec1d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk" Jul 28 01:25:51.292539 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:25:51.292448 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk" Jul 28 01:25:51.423323 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:25:51.423299 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk"] Jul 28 01:25:51.425195 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:25:51.425171 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dbcb09e_a463_427d_8e2a_0023c811ec1d.slice/crio-acd2f5207d617d7f7768479fddb859cf57d5de1ee612c942cbdd34295e851c0f WatchSource:0}: Error finding container acd2f5207d617d7f7768479fddb859cf57d5de1ee612c942cbdd34295e851c0f: Status 404 returned error can't find the container with id acd2f5207d617d7f7768479fddb859cf57d5de1ee612c942cbdd34295e851c0f Jul 28 01:25:51.846293 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:25:51.846259 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk" event={"ID":"7dbcb09e-a463-427d-8e2a-0023c811ec1d","Type":"ContainerStarted","Data":"acd2f5207d617d7f7768479fddb859cf57d5de1ee612c942cbdd34295e851c0f"} Jul 28 01:25:52.852447 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:25:52.852413 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk" event={"ID":"7dbcb09e-a463-427d-8e2a-0023c811ec1d","Type":"ContainerStarted","Data":"835e4f5c426d546e10972cb72b1c8f7cc770f98502733c45cf3d12f506bdc1e5"} Jul 28 01:25:52.869844 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:25:52.869793 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk" podStartSLOduration=1.6393323720000001 podStartE2EDuration="2.869778375s" totalImagesPullingTime="1.230446003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:25:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:25:52.868661999 +0000 UTC m=+610.303060313" watchObservedRunningTime="2026-07-28 01:25:52.869778375 +0000 UTC m=+610.304176689" Jul 28 01:26:01.293508 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:26:01.293470 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk" Jul 28 01:26:01.294003 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:26:01.293762 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk" Jul 28 01:26:01.306169 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:26:01.306144 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk" Jul 28 01:26:01.903340 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:26:01.903306 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk" Jul 28 01:26:48.891280 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:26:48.891244 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vswsx"] Jul 28 01:26:48.894550 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:26:48.894532 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vswsx" Jul 28 01:26:48.897245 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:26:48.897226 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 28 01:26:48.904410 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:26:48.904384 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vswsx"] Jul 28 01:26:48.969649 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:26:48.969620 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vswsx\" (UID: \"7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vswsx" Jul 28 01:26:48.969831 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:26:48.969656 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vswsx\" (UID: \"7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vswsx" Jul 28 01:26:48.969831 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:26:48.969675 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vswsx\" (UID: \"7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vswsx" Jul 28 01:26:48.969831 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:26:48.969702 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vswsx\" (UID: \"7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vswsx" Jul 28 01:26:48.969831 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:26:48.969728 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d56zk\" (UniqueName: \"kubernetes.io/projected/7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15-kube-api-access-d56zk\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vswsx\" (UID: \"7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vswsx" Jul 28 01:26:48.969831 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:26:48.969783 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vswsx\" (UID: \"7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vswsx" Jul 28 01:26:49.070355 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:26:49.070314 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vswsx\" (UID: \"7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vswsx" Jul 28 01:26:49.070523 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:26:49.070372 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vswsx\" (UID: \"7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vswsx" Jul 28 01:26:49.070523 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:26:49.070401 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vswsx\" (UID: \"7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vswsx" Jul 28 01:26:49.070523 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:26:49.070440 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vswsx\" (UID: \"7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vswsx" Jul 28 01:26:49.070685 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:26:49.070575 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d56zk\" (UniqueName: \"kubernetes.io/projected/7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15-kube-api-access-d56zk\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vswsx\" (UID: \"7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vswsx" Jul 28 01:26:49.070685 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:26:49.070655 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vswsx\" (UID: \"7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vswsx" Jul 28 01:26:49.070778 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:26:49.070756 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vswsx\" (UID: \"7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vswsx" Jul 28 01:26:49.070778 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:26:49.070770 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vswsx\" (UID: \"7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vswsx" Jul 28 01:26:49.071006 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:26:49.070984 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vswsx\" (UID: \"7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vswsx" Jul 28 01:26:49.072727 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:26:49.072701 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vswsx\" (UID: \"7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vswsx" Jul 28 01:26:49.072896 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:26:49.072880 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vswsx\" (UID: \"7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vswsx" Jul 28 01:26:49.078349 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:26:49.078318 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d56zk\" (UniqueName: \"kubernetes.io/projected/7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15-kube-api-access-d56zk\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vswsx\" (UID: \"7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vswsx" Jul 28 01:26:49.204322 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:26:49.204295 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vswsx" Jul 28 01:26:49.337088 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:26:49.334852 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vswsx"] Jul 28 01:26:49.338449 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:26:49.338425 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 01:26:50.130504 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:26:50.130470 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vswsx" event={"ID":"7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15","Type":"ContainerStarted","Data":"1e5ba454e02d088560893b8845926fb2b77d4bac6512e5d2b695644446e05016"} Jul 28 01:26:50.130979 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:26:50.130520 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vswsx" event={"ID":"7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15","Type":"ContainerStarted","Data":"7f1a90038b0f75aba935cc0736bedf07465865b8fdb916e1bdcc13b44abad92b"} Jul 28 01:26:50.147933 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:26:50.147885 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vswsx" podStartSLOduration=1.9711133570000001 podStartE2EDuration="2.147872189s" totalImagesPullingTime="176.758832ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:50.146357894 +0000 UTC m=+667.580756209" watchObservedRunningTime="2026-07-28 01:26:50.147872189 +0000 UTC m=+667.582270502" Jul 28 01:26:59.204542 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:26:59.204498 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vswsx" Jul 28 01:26:59.205072 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:26:59.204565 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vswsx" Jul 28 01:26:59.217519 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:26:59.217494 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vswsx" Jul 28 01:27:00.190257 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:27:00.190221 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vswsx" Jul 28 01:39:14.892895 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:14.892861 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4"] Jul 28 01:39:14.896091 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:14.896075 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4" Jul 28 01:39:14.898667 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:14.898650 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-4be075e6-kserve-self-signed-certs\"" Jul 28 01:39:14.904406 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:14.904384 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4"] Jul 28 01:39:15.003890 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:15.003855 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9f13065-1cae-4ca7-9027-e3dff8481aff-dshm\") pod \"e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4\" (UID: \"a9f13065-1cae-4ca7-9027-e3dff8481aff\") " pod="llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4" Jul 28 01:39:15.004078 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:15.003900 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9f13065-1cae-4ca7-9027-e3dff8481aff-tls-certs\") pod \"e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4\" (UID: \"a9f13065-1cae-4ca7-9027-e3dff8481aff\") " pod="llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4" Jul 28 01:39:15.004078 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:15.003923 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9f13065-1cae-4ca7-9027-e3dff8481aff-model-cache\") pod \"e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4\" (UID: \"a9f13065-1cae-4ca7-9027-e3dff8481aff\") " pod="llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4" Jul 28 01:39:15.004078 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:15.004019 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9f13065-1cae-4ca7-9027-e3dff8481aff-home\") pod \"e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4\" (UID: \"a9f13065-1cae-4ca7-9027-e3dff8481aff\") " pod="llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4" Jul 28 01:39:15.004242 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:15.004121 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62gfd\" (UniqueName: \"kubernetes.io/projected/a9f13065-1cae-4ca7-9027-e3dff8481aff-kube-api-access-62gfd\") pod \"e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4\" (UID: \"a9f13065-1cae-4ca7-9027-e3dff8481aff\") " pod="llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4" Jul 28 01:39:15.004242 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:15.004149 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9f13065-1cae-4ca7-9027-e3dff8481aff-tmp-dir\") pod \"e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4\" (UID: \"a9f13065-1cae-4ca7-9027-e3dff8481aff\") " pod="llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4" Jul 28 01:39:15.104911 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:15.104880 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9f13065-1cae-4ca7-9027-e3dff8481aff-tls-certs\") pod \"e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4\" (UID: \"a9f13065-1cae-4ca7-9027-e3dff8481aff\") " pod="llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4" Jul 28 01:39:15.104911 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:15.104912 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9f13065-1cae-4ca7-9027-e3dff8481aff-model-cache\") pod \"e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4\" (UID: \"a9f13065-1cae-4ca7-9027-e3dff8481aff\") " pod="llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4" Jul 28 01:39:15.105183 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:15.104933 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9f13065-1cae-4ca7-9027-e3dff8481aff-home\") pod \"e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4\" (UID: \"a9f13065-1cae-4ca7-9027-e3dff8481aff\") " pod="llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4" Jul 28 01:39:15.105183 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:15.104975 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62gfd\" (UniqueName: \"kubernetes.io/projected/a9f13065-1cae-4ca7-9027-e3dff8481aff-kube-api-access-62gfd\") pod \"e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4\" (UID: \"a9f13065-1cae-4ca7-9027-e3dff8481aff\") " pod="llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4" Jul 28 01:39:15.105183 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:15.104995 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9f13065-1cae-4ca7-9027-e3dff8481aff-tmp-dir\") pod \"e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4\" (UID: \"a9f13065-1cae-4ca7-9027-e3dff8481aff\") " pod="llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4" Jul 28 01:39:15.105183 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:15.105028 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9f13065-1cae-4ca7-9027-e3dff8481aff-dshm\") pod \"e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4\" (UID: \"a9f13065-1cae-4ca7-9027-e3dff8481aff\") " pod="llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4" Jul 28 01:39:15.105394 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:15.105326 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9f13065-1cae-4ca7-9027-e3dff8481aff-model-cache\") pod \"e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4\" (UID: \"a9f13065-1cae-4ca7-9027-e3dff8481aff\") " pod="llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4" Jul 28 01:39:15.105394 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:15.105365 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9f13065-1cae-4ca7-9027-e3dff8481aff-tmp-dir\") pod \"e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4\" (UID: \"a9f13065-1cae-4ca7-9027-e3dff8481aff\") " pod="llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4" Jul 28 01:39:15.105471 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:15.105460 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9f13065-1cae-4ca7-9027-e3dff8481aff-home\") pod \"e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4\" (UID: \"a9f13065-1cae-4ca7-9027-e3dff8481aff\") " pod="llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4" Jul 28 01:39:15.107337 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:15.107317 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9f13065-1cae-4ca7-9027-e3dff8481aff-dshm\") pod \"e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4\" (UID: \"a9f13065-1cae-4ca7-9027-e3dff8481aff\") " pod="llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4" Jul 28 01:39:15.107610 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:15.107592 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9f13065-1cae-4ca7-9027-e3dff8481aff-tls-certs\") pod \"e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4\" (UID: \"a9f13065-1cae-4ca7-9027-e3dff8481aff\") " pod="llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4" Jul 28 01:39:15.112780 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:15.112755 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62gfd\" (UniqueName: \"kubernetes.io/projected/a9f13065-1cae-4ca7-9027-e3dff8481aff-kube-api-access-62gfd\") pod \"e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4\" (UID: \"a9f13065-1cae-4ca7-9027-e3dff8481aff\") " pod="llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4" Jul 28 01:39:15.205871 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:15.205837 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4" Jul 28 01:39:15.333610 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:15.333577 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4"] Jul 28 01:39:15.335359 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:39:15.335323 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9f13065_1cae_4ca7_9027_e3dff8481aff.slice/crio-3962a3c379ecf52a2121050bfaa092186db14a6b4da877e51a918096b4c5bb86 WatchSource:0}: Error finding container 3962a3c379ecf52a2121050bfaa092186db14a6b4da877e51a918096b4c5bb86: Status 404 returned error can't find the container with id 3962a3c379ecf52a2121050bfaa092186db14a6b4da877e51a918096b4c5bb86 Jul 28 01:39:15.721718 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:15.721681 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4" event={"ID":"a9f13065-1cae-4ca7-9027-e3dff8481aff","Type":"ContainerStarted","Data":"96efbb9db1ff5a6341925f39f836b03dbfaeaa4b3d1210a8255e7bd6233a5f8b"} Jul 28 01:39:15.721718 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:15.721724 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4" event={"ID":"a9f13065-1cae-4ca7-9027-e3dff8481aff","Type":"ContainerStarted","Data":"3962a3c379ecf52a2121050bfaa092186db14a6b4da877e51a918096b4c5bb86"} Jul 28 01:39:15.740413 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:15.740358 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4" podStartSLOduration=1.740339651 podStartE2EDuration="1.740339651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:39:15.738505274 +0000 UTC m=+1413.172903590" watchObservedRunningTime="2026-07-28 01:39:15.740339651 +0000 UTC m=+1413.174737965" Jul 28 01:39:25.206549 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:25.206510 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4" Jul 28 01:39:25.206944 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:25.206733 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4" Jul 28 01:39:25.219250 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:25.219225 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4" Jul 28 01:39:25.781768 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:25.781736 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4" Jul 28 01:39:32.590458 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:32.590423 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4"] Jul 28 01:39:47.609764 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:47.609706 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4" podUID="a9f13065-1cae-4ca7-9027-e3dff8481aff" containerName="main" containerID="cri-o://96efbb9db1ff5a6341925f39f836b03dbfaeaa4b3d1210a8255e7bd6233a5f8b" gracePeriod=45 Jul 28 01:39:47.847286 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:47.847265 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4" Jul 28 01:39:47.877332 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:47.877268 2565 generic.go:356] "Generic (PLEG): container finished" podID="a9f13065-1cae-4ca7-9027-e3dff8481aff" containerID="96efbb9db1ff5a6341925f39f836b03dbfaeaa4b3d1210a8255e7bd6233a5f8b" exitCode=0 Jul 28 01:39:47.877332 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:47.877305 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4" Jul 28 01:39:47.877526 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:47.877353 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4" event={"ID":"a9f13065-1cae-4ca7-9027-e3dff8481aff","Type":"ContainerDied","Data":"96efbb9db1ff5a6341925f39f836b03dbfaeaa4b3d1210a8255e7bd6233a5f8b"} Jul 28 01:39:47.877526 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:47.877399 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4" event={"ID":"a9f13065-1cae-4ca7-9027-e3dff8481aff","Type":"ContainerDied","Data":"3962a3c379ecf52a2121050bfaa092186db14a6b4da877e51a918096b4c5bb86"} Jul 28 01:39:47.877526 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:47.877424 2565 scope.go:117] "RemoveContainer" containerID="96efbb9db1ff5a6341925f39f836b03dbfaeaa4b3d1210a8255e7bd6233a5f8b" Jul 28 01:39:47.885690 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:47.885675 2565 scope.go:117] "RemoveContainer" containerID="96efbb9db1ff5a6341925f39f836b03dbfaeaa4b3d1210a8255e7bd6233a5f8b" Jul 28 01:39:47.885909 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:39:47.885887 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96efbb9db1ff5a6341925f39f836b03dbfaeaa4b3d1210a8255e7bd6233a5f8b\": container with ID starting with 96efbb9db1ff5a6341925f39f836b03dbfaeaa4b3d1210a8255e7bd6233a5f8b not found: ID does not exist" containerID="96efbb9db1ff5a6341925f39f836b03dbfaeaa4b3d1210a8255e7bd6233a5f8b" Jul 28 01:39:47.886055 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:47.885915 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96efbb9db1ff5a6341925f39f836b03dbfaeaa4b3d1210a8255e7bd6233a5f8b"} err="failed to get container status \"96efbb9db1ff5a6341925f39f836b03dbfaeaa4b3d1210a8255e7bd6233a5f8b\": rpc error: code = NotFound desc = could not find container \"96efbb9db1ff5a6341925f39f836b03dbfaeaa4b3d1210a8255e7bd6233a5f8b\": container with ID starting with 96efbb9db1ff5a6341925f39f836b03dbfaeaa4b3d1210a8255e7bd6233a5f8b not found: ID does not exist" Jul 28 01:39:47.901604 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:47.901585 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9f13065-1cae-4ca7-9027-e3dff8481aff-tmp-dir\") pod \"a9f13065-1cae-4ca7-9027-e3dff8481aff\" (UID: \"a9f13065-1cae-4ca7-9027-e3dff8481aff\") " Jul 28 01:39:47.901696 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:47.901613 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9f13065-1cae-4ca7-9027-e3dff8481aff-dshm\") pod \"a9f13065-1cae-4ca7-9027-e3dff8481aff\" (UID: \"a9f13065-1cae-4ca7-9027-e3dff8481aff\") " Jul 28 01:39:47.901696 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:47.901661 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9f13065-1cae-4ca7-9027-e3dff8481aff-model-cache\") pod \"a9f13065-1cae-4ca7-9027-e3dff8481aff\" (UID: \"a9f13065-1cae-4ca7-9027-e3dff8481aff\") " Jul 28 01:39:47.901696 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:47.901691 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9f13065-1cae-4ca7-9027-e3dff8481aff-home\") pod \"a9f13065-1cae-4ca7-9027-e3dff8481aff\" (UID: \"a9f13065-1cae-4ca7-9027-e3dff8481aff\") " Jul 28 01:39:47.901825 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:47.901710 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62gfd\" (UniqueName: \"kubernetes.io/projected/a9f13065-1cae-4ca7-9027-e3dff8481aff-kube-api-access-62gfd\") pod \"a9f13065-1cae-4ca7-9027-e3dff8481aff\" (UID: \"a9f13065-1cae-4ca7-9027-e3dff8481aff\") " Jul 28 01:39:47.901825 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:47.901764 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9f13065-1cae-4ca7-9027-e3dff8481aff-tls-certs\") pod \"a9f13065-1cae-4ca7-9027-e3dff8481aff\" (UID: \"a9f13065-1cae-4ca7-9027-e3dff8481aff\") " Jul 28 01:39:47.901908 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:47.901836 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9f13065-1cae-4ca7-9027-e3dff8481aff-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a9f13065-1cae-4ca7-9027-e3dff8481aff" (UID: "a9f13065-1cae-4ca7-9027-e3dff8481aff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:39:47.901958 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:47.901943 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9f13065-1cae-4ca7-9027-e3dff8481aff-model-cache" (OuterVolumeSpecName: "model-cache") pod "a9f13065-1cae-4ca7-9027-e3dff8481aff" (UID: "a9f13065-1cae-4ca7-9027-e3dff8481aff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:39:47.902010 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:47.901960 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9f13065-1cae-4ca7-9027-e3dff8481aff-home" (OuterVolumeSpecName: "home") pod "a9f13065-1cae-4ca7-9027-e3dff8481aff" (UID: "a9f13065-1cae-4ca7-9027-e3dff8481aff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:39:47.902092 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:47.902050 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9f13065-1cae-4ca7-9027-e3dff8481aff-tmp-dir\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:39:47.903713 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:47.903692 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9f13065-1cae-4ca7-9027-e3dff8481aff-dshm" (OuterVolumeSpecName: "dshm") pod "a9f13065-1cae-4ca7-9027-e3dff8481aff" (UID: "a9f13065-1cae-4ca7-9027-e3dff8481aff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:39:47.903807 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:47.903695 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9f13065-1cae-4ca7-9027-e3dff8481aff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a9f13065-1cae-4ca7-9027-e3dff8481aff" (UID: "a9f13065-1cae-4ca7-9027-e3dff8481aff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:39:47.903807 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:47.903755 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9f13065-1cae-4ca7-9027-e3dff8481aff-kube-api-access-62gfd" (OuterVolumeSpecName: "kube-api-access-62gfd") pod "a9f13065-1cae-4ca7-9027-e3dff8481aff" (UID: "a9f13065-1cae-4ca7-9027-e3dff8481aff"). InnerVolumeSpecName "kube-api-access-62gfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:39:48.003246 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:48.003213 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9f13065-1cae-4ca7-9027-e3dff8481aff-dshm\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:39:48.003246 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:48.003240 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9f13065-1cae-4ca7-9027-e3dff8481aff-model-cache\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:39:48.003246 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:48.003249 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9f13065-1cae-4ca7-9027-e3dff8481aff-home\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:39:48.003461 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:48.003259 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62gfd\" (UniqueName: \"kubernetes.io/projected/a9f13065-1cae-4ca7-9027-e3dff8481aff-kube-api-access-62gfd\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:39:48.003461 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:48.003268 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9f13065-1cae-4ca7-9027-e3dff8481aff-tls-certs\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:39:48.198737 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:48.198710 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4"] Jul 28 01:39:48.204276 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:48.204243 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-4be075e6-kserve-bd59796dc-j58f4"] Jul 28 01:39:49.100853 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:39:49.100824 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9f13065-1cae-4ca7-9027-e3dff8481aff" path="/var/lib/kubelet/pods/a9f13065-1cae-4ca7-9027-e3dff8481aff/volumes" Jul 28 01:43:20.900944 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:20.900904 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9dc1294f-5776c85954-h97hm"] Jul 28 01:43:20.903417 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:20.901410 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9f13065-1cae-4ca7-9027-e3dff8481aff" containerName="main" Jul 28 01:43:20.903417 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:20.901430 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9f13065-1cae-4ca7-9027-e3dff8481aff" containerName="main" Jul 28 01:43:20.903417 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:20.901516 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9f13065-1cae-4ca7-9027-e3dff8481aff" containerName="main" Jul 28 01:43:20.904414 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:20.904395 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dc1294f-5776c85954-h97hm" Jul 28 01:43:20.907084 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:20.907065 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9dc1294f-serving-cert\"" Jul 28 01:43:20.907220 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:20.907198 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-62vvq\"" Jul 28 01:43:20.908301 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:20.908285 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 28 01:43:20.908301 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:20.908292 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 01:43:20.916811 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:20.916786 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9dc1294f-5776c85954-h97hm"] Jul 28 01:43:20.997764 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:20.997729 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7knl\" (UniqueName: \"kubernetes.io/projected/78bb12c3-d61e-4cc9-b86a-5f7aae199cf3-kube-api-access-c7knl\") pod \"maas-api-e2e-ait-9dc1294f-5776c85954-h97hm\" (UID: \"78bb12c3-d61e-4cc9-b86a-5f7aae199cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dc1294f-5776c85954-h97hm" Jul 28 01:43:20.997939 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:20.997863 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78bb12c3-d61e-4cc9-b86a-5f7aae199cf3-openshift-service-ca\") pod \"maas-api-e2e-ait-9dc1294f-5776c85954-h97hm\" (UID: \"78bb12c3-d61e-4cc9-b86a-5f7aae199cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dc1294f-5776c85954-h97hm" Jul 28 01:43:20.997939 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:20.997934 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78bb12c3-d61e-4cc9-b86a-5f7aae199cf3-maas-api-tls\") pod \"maas-api-e2e-ait-9dc1294f-5776c85954-h97hm\" (UID: \"78bb12c3-d61e-4cc9-b86a-5f7aae199cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dc1294f-5776c85954-h97hm" Jul 28 01:43:21.098712 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:21.098678 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7knl\" (UniqueName: \"kubernetes.io/projected/78bb12c3-d61e-4cc9-b86a-5f7aae199cf3-kube-api-access-c7knl\") pod \"maas-api-e2e-ait-9dc1294f-5776c85954-h97hm\" (UID: \"78bb12c3-d61e-4cc9-b86a-5f7aae199cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dc1294f-5776c85954-h97hm" Jul 28 01:43:21.098900 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:21.098724 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78bb12c3-d61e-4cc9-b86a-5f7aae199cf3-openshift-service-ca\") pod \"maas-api-e2e-ait-9dc1294f-5776c85954-h97hm\" (UID: \"78bb12c3-d61e-4cc9-b86a-5f7aae199cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dc1294f-5776c85954-h97hm" Jul 28 01:43:21.098900 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:21.098753 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78bb12c3-d61e-4cc9-b86a-5f7aae199cf3-maas-api-tls\") pod \"maas-api-e2e-ait-9dc1294f-5776c85954-h97hm\" (UID: \"78bb12c3-d61e-4cc9-b86a-5f7aae199cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dc1294f-5776c85954-h97hm" Jul 28 01:43:21.098900 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:43:21.098846 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-9dc1294f-serving-cert: secret "maas-api-e2e-ait-9dc1294f-serving-cert" not found Jul 28 01:43:21.098900 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:43:21.098890 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78bb12c3-d61e-4cc9-b86a-5f7aae199cf3-maas-api-tls podName:78bb12c3-d61e-4cc9-b86a-5f7aae199cf3 nodeName:}" failed. No retries permitted until 2026-07-28 01:43:21.598876033 +0000 UTC m=+1659.033274328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/78bb12c3-d61e-4cc9-b86a-5f7aae199cf3-maas-api-tls") pod "maas-api-e2e-ait-9dc1294f-5776c85954-h97hm" (UID: "78bb12c3-d61e-4cc9-b86a-5f7aae199cf3") : secret "maas-api-e2e-ait-9dc1294f-serving-cert" not found Jul 28 01:43:21.099441 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:21.099417 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78bb12c3-d61e-4cc9-b86a-5f7aae199cf3-openshift-service-ca\") pod \"maas-api-e2e-ait-9dc1294f-5776c85954-h97hm\" (UID: \"78bb12c3-d61e-4cc9-b86a-5f7aae199cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dc1294f-5776c85954-h97hm" Jul 28 01:43:21.112205 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:21.112178 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7knl\" (UniqueName: \"kubernetes.io/projected/78bb12c3-d61e-4cc9-b86a-5f7aae199cf3-kube-api-access-c7knl\") pod \"maas-api-e2e-ait-9dc1294f-5776c85954-h97hm\" (UID: \"78bb12c3-d61e-4cc9-b86a-5f7aae199cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dc1294f-5776c85954-h97hm" Jul 28 01:43:21.604888 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:21.604854 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78bb12c3-d61e-4cc9-b86a-5f7aae199cf3-maas-api-tls\") pod \"maas-api-e2e-ait-9dc1294f-5776c85954-h97hm\" (UID: \"78bb12c3-d61e-4cc9-b86a-5f7aae199cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dc1294f-5776c85954-h97hm" Jul 28 01:43:21.607273 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:21.607244 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78bb12c3-d61e-4cc9-b86a-5f7aae199cf3-maas-api-tls\") pod \"maas-api-e2e-ait-9dc1294f-5776c85954-h97hm\" (UID: \"78bb12c3-d61e-4cc9-b86a-5f7aae199cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dc1294f-5776c85954-h97hm" Jul 28 01:43:21.815050 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:21.815015 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dc1294f-5776c85954-h97hm" Jul 28 01:43:21.948839 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:21.948661 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9dc1294f-5776c85954-h97hm"] Jul 28 01:43:21.951109 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:43:21.951082 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78bb12c3_d61e_4cc9_b86a_5f7aae199cf3.slice/crio-2ba78553826f8d0a26684b80321a6115e973059bffdaddd47debfbb4045cc4f2 WatchSource:0}: Error finding container 2ba78553826f8d0a26684b80321a6115e973059bffdaddd47debfbb4045cc4f2: Status 404 returned error can't find the container with id 2ba78553826f8d0a26684b80321a6115e973059bffdaddd47debfbb4045cc4f2 Jul 28 01:43:21.952558 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:21.952541 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 01:43:22.901888 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:22.901843 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dc1294f-5776c85954-h97hm" event={"ID":"78bb12c3-d61e-4cc9-b86a-5f7aae199cf3","Type":"ContainerStarted","Data":"2ba78553826f8d0a26684b80321a6115e973059bffdaddd47debfbb4045cc4f2"} Jul 28 01:43:24.913737 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:24.913702 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dc1294f-5776c85954-h97hm" event={"ID":"78bb12c3-d61e-4cc9-b86a-5f7aae199cf3","Type":"ContainerStarted","Data":"ef96e8707ed16d211f40b002d76ac030528396c1bdf5cd3b1a364713ad9e3c14"} Jul 28 01:43:24.914206 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:24.913807 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dc1294f-5776c85954-h97hm" Jul 28 01:43:24.933206 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:24.933163 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dc1294f-5776c85954-h97hm" podStartSLOduration=2.649363079 podStartE2EDuration="4.933149177s" totalImagesPullingTime="2.283786098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:43:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:43:24.931218064 +0000 UTC m=+1662.365616378" watchObservedRunningTime="2026-07-28 01:43:24.933149177 +0000 UTC m=+1662.367547539" Jul 28 01:43:30.924680 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:30.924646 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dc1294f-5776c85954-h97hm" Jul 28 01:43:56.774920 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:56.774884 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9dc1294f-5776c85954-h97hm"] Jul 28 01:43:56.775360 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:56.775154 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dc1294f-5776c85954-h97hm" podUID="78bb12c3-d61e-4cc9-b86a-5f7aae199cf3" containerName="maas-api" containerID="cri-o://ef96e8707ed16d211f40b002d76ac030528396c1bdf5cd3b1a364713ad9e3c14" gracePeriod=30 Jul 28 01:43:57.016442 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:57.016415 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dc1294f-5776c85954-h97hm" Jul 28 01:43:57.063551 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:57.063473 2565 generic.go:356] "Generic (PLEG): container finished" podID="78bb12c3-d61e-4cc9-b86a-5f7aae199cf3" containerID="ef96e8707ed16d211f40b002d76ac030528396c1bdf5cd3b1a364713ad9e3c14" exitCode=0 Jul 28 01:43:57.063551 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:57.063522 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dc1294f-5776c85954-h97hm" Jul 28 01:43:57.063712 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:57.063555 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dc1294f-5776c85954-h97hm" event={"ID":"78bb12c3-d61e-4cc9-b86a-5f7aae199cf3","Type":"ContainerDied","Data":"ef96e8707ed16d211f40b002d76ac030528396c1bdf5cd3b1a364713ad9e3c14"} Jul 28 01:43:57.063712 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:57.063607 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dc1294f-5776c85954-h97hm" event={"ID":"78bb12c3-d61e-4cc9-b86a-5f7aae199cf3","Type":"ContainerDied","Data":"2ba78553826f8d0a26684b80321a6115e973059bffdaddd47debfbb4045cc4f2"} Jul 28 01:43:57.063712 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:57.063629 2565 scope.go:117] "RemoveContainer" containerID="ef96e8707ed16d211f40b002d76ac030528396c1bdf5cd3b1a364713ad9e3c14" Jul 28 01:43:57.071744 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:57.071729 2565 scope.go:117] "RemoveContainer" containerID="ef96e8707ed16d211f40b002d76ac030528396c1bdf5cd3b1a364713ad9e3c14" Jul 28 01:43:57.072001 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:43:57.071984 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef96e8707ed16d211f40b002d76ac030528396c1bdf5cd3b1a364713ad9e3c14\": container with ID starting with ef96e8707ed16d211f40b002d76ac030528396c1bdf5cd3b1a364713ad9e3c14 not found: ID does not exist" containerID="ef96e8707ed16d211f40b002d76ac030528396c1bdf5cd3b1a364713ad9e3c14" Jul 28 01:43:57.072076 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:57.072009 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef96e8707ed16d211f40b002d76ac030528396c1bdf5cd3b1a364713ad9e3c14"} err="failed to get container status \"ef96e8707ed16d211f40b002d76ac030528396c1bdf5cd3b1a364713ad9e3c14\": rpc error: code = NotFound desc = could not find container \"ef96e8707ed16d211f40b002d76ac030528396c1bdf5cd3b1a364713ad9e3c14\": container with ID starting with ef96e8707ed16d211f40b002d76ac030528396c1bdf5cd3b1a364713ad9e3c14 not found: ID does not exist" Jul 28 01:43:57.114016 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:57.113995 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78bb12c3-d61e-4cc9-b86a-5f7aae199cf3-openshift-service-ca\") pod \"78bb12c3-d61e-4cc9-b86a-5f7aae199cf3\" (UID: \"78bb12c3-d61e-4cc9-b86a-5f7aae199cf3\") " Jul 28 01:43:57.114155 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:57.114080 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7knl\" (UniqueName: \"kubernetes.io/projected/78bb12c3-d61e-4cc9-b86a-5f7aae199cf3-kube-api-access-c7knl\") pod \"78bb12c3-d61e-4cc9-b86a-5f7aae199cf3\" (UID: \"78bb12c3-d61e-4cc9-b86a-5f7aae199cf3\") " Jul 28 01:43:57.114155 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:57.114103 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78bb12c3-d61e-4cc9-b86a-5f7aae199cf3-maas-api-tls\") pod \"78bb12c3-d61e-4cc9-b86a-5f7aae199cf3\" (UID: \"78bb12c3-d61e-4cc9-b86a-5f7aae199cf3\") " Jul 28 01:43:57.114352 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:57.114327 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78bb12c3-d61e-4cc9-b86a-5f7aae199cf3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "78bb12c3-d61e-4cc9-b86a-5f7aae199cf3" (UID: "78bb12c3-d61e-4cc9-b86a-5f7aae199cf3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:43:57.114507 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:57.114488 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78bb12c3-d61e-4cc9-b86a-5f7aae199cf3-openshift-service-ca\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:43:57.116227 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:57.116207 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78bb12c3-d61e-4cc9-b86a-5f7aae199cf3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "78bb12c3-d61e-4cc9-b86a-5f7aae199cf3" (UID: "78bb12c3-d61e-4cc9-b86a-5f7aae199cf3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:43:57.116315 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:57.116277 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78bb12c3-d61e-4cc9-b86a-5f7aae199cf3-kube-api-access-c7knl" (OuterVolumeSpecName: "kube-api-access-c7knl") pod "78bb12c3-d61e-4cc9-b86a-5f7aae199cf3" (UID: "78bb12c3-d61e-4cc9-b86a-5f7aae199cf3"). InnerVolumeSpecName "kube-api-access-c7knl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:43:57.215367 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:57.215338 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7knl\" (UniqueName: \"kubernetes.io/projected/78bb12c3-d61e-4cc9-b86a-5f7aae199cf3-kube-api-access-c7knl\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:43:57.215367 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:57.215363 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78bb12c3-d61e-4cc9-b86a-5f7aae199cf3-maas-api-tls\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:43:57.386142 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:57.386114 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9dc1294f-5776c85954-h97hm"] Jul 28 01:43:57.389969 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:57.389943 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9dc1294f-5776c85954-h97hm"] Jul 28 01:43:59.101025 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:43:59.100988 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78bb12c3-d61e-4cc9-b86a-5f7aae199cf3" path="/var/lib/kubelet/pods/78bb12c3-d61e-4cc9-b86a-5f7aae199cf3/volumes" Jul 28 01:44:05.488112 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:05.488011 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e887be66-75f4f6845d-t866g"] Jul 28 01:44:05.488669 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:05.488494 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78bb12c3-d61e-4cc9-b86a-5f7aae199cf3" containerName="maas-api" Jul 28 01:44:05.488669 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:05.488511 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="78bb12c3-d61e-4cc9-b86a-5f7aae199cf3" containerName="maas-api" Jul 28 01:44:05.488669 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:05.488564 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="78bb12c3-d61e-4cc9-b86a-5f7aae199cf3" containerName="maas-api" Jul 28 01:44:05.491660 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:05.491635 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e887be66-75f4f6845d-t866g" Jul 28 01:44:05.495188 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:05.495169 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 28 01:44:05.495419 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:05.495402 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-e887be66-serving-cert\"" Jul 28 01:44:05.496637 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:05.496619 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 01:44:05.496741 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:05.496621 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-62vvq\"" Jul 28 01:44:05.500675 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:05.500657 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e887be66-75f4f6845d-t866g"] Jul 28 01:44:05.588991 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:05.588954 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsglk\" (UniqueName: \"kubernetes.io/projected/263d7130-eeaa-4021-a7ca-f226b6f3f625-kube-api-access-tsglk\") pod \"maas-api-e2e-migrate-e887be66-75f4f6845d-t866g\" (UID: \"263d7130-eeaa-4021-a7ca-f226b6f3f625\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e887be66-75f4f6845d-t866g" Jul 28 01:44:05.589191 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:05.589017 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/263d7130-eeaa-4021-a7ca-f226b6f3f625-maas-api-tls\") pod \"maas-api-e2e-migrate-e887be66-75f4f6845d-t866g\" (UID: \"263d7130-eeaa-4021-a7ca-f226b6f3f625\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e887be66-75f4f6845d-t866g" Jul 28 01:44:05.589191 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:05.589081 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/263d7130-eeaa-4021-a7ca-f226b6f3f625-openshift-service-ca\") pod \"maas-api-e2e-migrate-e887be66-75f4f6845d-t866g\" (UID: \"263d7130-eeaa-4021-a7ca-f226b6f3f625\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e887be66-75f4f6845d-t866g" Jul 28 01:44:05.690109 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:05.690079 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsglk\" (UniqueName: \"kubernetes.io/projected/263d7130-eeaa-4021-a7ca-f226b6f3f625-kube-api-access-tsglk\") pod \"maas-api-e2e-migrate-e887be66-75f4f6845d-t866g\" (UID: \"263d7130-eeaa-4021-a7ca-f226b6f3f625\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e887be66-75f4f6845d-t866g" Jul 28 01:44:05.690248 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:05.690122 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/263d7130-eeaa-4021-a7ca-f226b6f3f625-maas-api-tls\") pod \"maas-api-e2e-migrate-e887be66-75f4f6845d-t866g\" (UID: \"263d7130-eeaa-4021-a7ca-f226b6f3f625\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e887be66-75f4f6845d-t866g" Jul 28 01:44:05.690248 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:05.690203 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/263d7130-eeaa-4021-a7ca-f226b6f3f625-openshift-service-ca\") pod \"maas-api-e2e-migrate-e887be66-75f4f6845d-t866g\" (UID: \"263d7130-eeaa-4021-a7ca-f226b6f3f625\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e887be66-75f4f6845d-t866g" Jul 28 01:44:05.690815 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:05.690799 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/263d7130-eeaa-4021-a7ca-f226b6f3f625-openshift-service-ca\") pod \"maas-api-e2e-migrate-e887be66-75f4f6845d-t866g\" (UID: \"263d7130-eeaa-4021-a7ca-f226b6f3f625\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e887be66-75f4f6845d-t866g" Jul 28 01:44:05.692511 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:05.692491 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/263d7130-eeaa-4021-a7ca-f226b6f3f625-maas-api-tls\") pod \"maas-api-e2e-migrate-e887be66-75f4f6845d-t866g\" (UID: \"263d7130-eeaa-4021-a7ca-f226b6f3f625\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e887be66-75f4f6845d-t866g" Jul 28 01:44:05.698045 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:05.698014 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsglk\" (UniqueName: \"kubernetes.io/projected/263d7130-eeaa-4021-a7ca-f226b6f3f625-kube-api-access-tsglk\") pod \"maas-api-e2e-migrate-e887be66-75f4f6845d-t866g\" (UID: \"263d7130-eeaa-4021-a7ca-f226b6f3f625\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e887be66-75f4f6845d-t866g" Jul 28 01:44:05.802584 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:05.802516 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e887be66-75f4f6845d-t866g" Jul 28 01:44:05.937023 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:05.936994 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e887be66-75f4f6845d-t866g"] Jul 28 01:44:05.939446 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:44:05.939420 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod263d7130_eeaa_4021_a7ca_f226b6f3f625.slice/crio-eab51ca7e61812d5d785075c91d7f017fceb24ced3e8acb3d1736684b4d5764e WatchSource:0}: Error finding container eab51ca7e61812d5d785075c91d7f017fceb24ced3e8acb3d1736684b4d5764e: Status 404 returned error can't find the container with id eab51ca7e61812d5d785075c91d7f017fceb24ced3e8acb3d1736684b4d5764e Jul 28 01:44:06.115567 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:06.115477 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e887be66-75f4f6845d-t866g" event={"ID":"263d7130-eeaa-4021-a7ca-f226b6f3f625","Type":"ContainerStarted","Data":"eab51ca7e61812d5d785075c91d7f017fceb24ced3e8acb3d1736684b4d5764e"} Jul 28 01:44:07.121717 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:07.121681 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e887be66-75f4f6845d-t866g" event={"ID":"263d7130-eeaa-4021-a7ca-f226b6f3f625","Type":"ContainerStarted","Data":"5a7437f8b9d586ef282696d94dcdace5a236ebb4f04866e6de542ff9a4a294b3"} Jul 28 01:44:07.122222 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:07.121793 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e887be66-75f4f6845d-t866g" Jul 28 01:44:07.140689 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:07.140636 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e887be66-75f4f6845d-t866g" podStartSLOduration=1.7574318089999998 podStartE2EDuration="2.140621088s" totalImagesPullingTime="383.189279ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:44:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:44:07.139696879 +0000 UTC m=+1704.574095191" watchObservedRunningTime="2026-07-28 01:44:07.140621088 +0000 UTC m=+1704.575019402" Jul 28 01:44:13.131552 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:13.131525 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e887be66-75f4f6845d-t866g" Jul 28 01:44:40.695750 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:40.695717 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e887be66-75f4f6845d-t866g"] Jul 28 01:44:40.696344 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:40.695946 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e887be66-75f4f6845d-t866g" podUID="263d7130-eeaa-4021-a7ca-f226b6f3f625" containerName="maas-api" containerID="cri-o://5a7437f8b9d586ef282696d94dcdace5a236ebb4f04866e6de542ff9a4a294b3" gracePeriod=30 Jul 28 01:44:40.942689 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:40.942662 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e887be66-75f4f6845d-t866g" Jul 28 01:44:40.998938 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:40.998847 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/263d7130-eeaa-4021-a7ca-f226b6f3f625-maas-api-tls\") pod \"263d7130-eeaa-4021-a7ca-f226b6f3f625\" (UID: \"263d7130-eeaa-4021-a7ca-f226b6f3f625\") " Jul 28 01:44:40.998938 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:40.998899 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/263d7130-eeaa-4021-a7ca-f226b6f3f625-openshift-service-ca\") pod \"263d7130-eeaa-4021-a7ca-f226b6f3f625\" (UID: \"263d7130-eeaa-4021-a7ca-f226b6f3f625\") " Jul 28 01:44:40.999156 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:40.998949 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsglk\" (UniqueName: \"kubernetes.io/projected/263d7130-eeaa-4021-a7ca-f226b6f3f625-kube-api-access-tsglk\") pod \"263d7130-eeaa-4021-a7ca-f226b6f3f625\" (UID: \"263d7130-eeaa-4021-a7ca-f226b6f3f625\") " Jul 28 01:44:40.999337 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:40.999312 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/263d7130-eeaa-4021-a7ca-f226b6f3f625-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "263d7130-eeaa-4021-a7ca-f226b6f3f625" (UID: "263d7130-eeaa-4021-a7ca-f226b6f3f625"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:44:41.001114 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:41.001086 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/263d7130-eeaa-4021-a7ca-f226b6f3f625-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "263d7130-eeaa-4021-a7ca-f226b6f3f625" (UID: "263d7130-eeaa-4021-a7ca-f226b6f3f625"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:44:41.001224 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:41.001111 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/263d7130-eeaa-4021-a7ca-f226b6f3f625-kube-api-access-tsglk" (OuterVolumeSpecName: "kube-api-access-tsglk") pod "263d7130-eeaa-4021-a7ca-f226b6f3f625" (UID: "263d7130-eeaa-4021-a7ca-f226b6f3f625"). InnerVolumeSpecName "kube-api-access-tsglk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:44:41.100362 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:41.100328 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsglk\" (UniqueName: \"kubernetes.io/projected/263d7130-eeaa-4021-a7ca-f226b6f3f625-kube-api-access-tsglk\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:44:41.100362 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:41.100359 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/263d7130-eeaa-4021-a7ca-f226b6f3f625-maas-api-tls\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:44:41.100560 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:41.100372 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/263d7130-eeaa-4021-a7ca-f226b6f3f625-openshift-service-ca\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:44:41.285389 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:41.285297 2565 generic.go:356] "Generic (PLEG): container finished" podID="263d7130-eeaa-4021-a7ca-f226b6f3f625" containerID="5a7437f8b9d586ef282696d94dcdace5a236ebb4f04866e6de542ff9a4a294b3" exitCode=0 Jul 28 01:44:41.285389 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:41.285348 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e887be66-75f4f6845d-t866g" Jul 28 01:44:41.285575 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:41.285405 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e887be66-75f4f6845d-t866g" event={"ID":"263d7130-eeaa-4021-a7ca-f226b6f3f625","Type":"ContainerDied","Data":"5a7437f8b9d586ef282696d94dcdace5a236ebb4f04866e6de542ff9a4a294b3"} Jul 28 01:44:41.285575 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:41.285460 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e887be66-75f4f6845d-t866g" event={"ID":"263d7130-eeaa-4021-a7ca-f226b6f3f625","Type":"ContainerDied","Data":"eab51ca7e61812d5d785075c91d7f017fceb24ced3e8acb3d1736684b4d5764e"} Jul 28 01:44:41.285575 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:41.285484 2565 scope.go:117] "RemoveContainer" containerID="5a7437f8b9d586ef282696d94dcdace5a236ebb4f04866e6de542ff9a4a294b3" Jul 28 01:44:41.293913 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:41.293896 2565 scope.go:117] "RemoveContainer" containerID="5a7437f8b9d586ef282696d94dcdace5a236ebb4f04866e6de542ff9a4a294b3" Jul 28 01:44:41.294224 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:44:41.294197 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a7437f8b9d586ef282696d94dcdace5a236ebb4f04866e6de542ff9a4a294b3\": container with ID starting with 5a7437f8b9d586ef282696d94dcdace5a236ebb4f04866e6de542ff9a4a294b3 not found: ID does not exist" containerID="5a7437f8b9d586ef282696d94dcdace5a236ebb4f04866e6de542ff9a4a294b3" Jul 28 01:44:41.294306 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:41.294230 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a7437f8b9d586ef282696d94dcdace5a236ebb4f04866e6de542ff9a4a294b3"} err="failed to get container status \"5a7437f8b9d586ef282696d94dcdace5a236ebb4f04866e6de542ff9a4a294b3\": rpc error: code = NotFound desc = could not find container \"5a7437f8b9d586ef282696d94dcdace5a236ebb4f04866e6de542ff9a4a294b3\": container with ID starting with 5a7437f8b9d586ef282696d94dcdace5a236ebb4f04866e6de542ff9a4a294b3 not found: ID does not exist" Jul 28 01:44:41.302210 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:41.302183 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e887be66-75f4f6845d-t866g"] Jul 28 01:44:41.305993 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:41.305970 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e887be66-75f4f6845d-t866g"] Jul 28 01:44:43.101097 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:43.101065 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="263d7130-eeaa-4021-a7ca-f226b6f3f625" path="/var/lib/kubelet/pods/263d7130-eeaa-4021-a7ca-f226b6f3f625/volumes" Jul 28 01:44:48.411516 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:48.411482 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd"] Jul 28 01:44:48.411880 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:48.411822 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="263d7130-eeaa-4021-a7ca-f226b6f3f625" containerName="maas-api" Jul 28 01:44:48.411880 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:48.411840 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="263d7130-eeaa-4021-a7ca-f226b6f3f625" containerName="maas-api" Jul 28 01:44:48.411986 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:48.411917 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="263d7130-eeaa-4021-a7ca-f226b6f3f625" containerName="maas-api" Jul 28 01:44:48.414917 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:48.414897 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd" Jul 28 01:44:48.418397 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:48.418375 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1c712754-serving-cert\"" Jul 28 01:44:48.418531 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:48.418426 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 28 01:44:48.418531 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:48.418478 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 01:44:48.418531 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:48.418512 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-62vvq\"" Jul 28 01:44:48.424638 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:48.424604 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd"] Jul 28 01:44:48.464334 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:48.464303 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns9hp\" (UniqueName: \"kubernetes.io/projected/2896a029-6b06-4771-bb18-35a1464bde6e-kube-api-access-ns9hp\") pod \"maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd\" (UID: \"2896a029-6b06-4771-bb18-35a1464bde6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd" Jul 28 01:44:48.464525 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:48.464350 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2896a029-6b06-4771-bb18-35a1464bde6e-openshift-service-ca\") pod \"maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd\" (UID: \"2896a029-6b06-4771-bb18-35a1464bde6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd" Jul 28 01:44:48.464525 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:48.464489 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2896a029-6b06-4771-bb18-35a1464bde6e-maas-api-tls\") pod \"maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd\" (UID: \"2896a029-6b06-4771-bb18-35a1464bde6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd" Jul 28 01:44:48.565964 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:48.565926 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2896a029-6b06-4771-bb18-35a1464bde6e-maas-api-tls\") pod \"maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd\" (UID: \"2896a029-6b06-4771-bb18-35a1464bde6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd" Jul 28 01:44:48.566210 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:48.565978 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns9hp\" (UniqueName: \"kubernetes.io/projected/2896a029-6b06-4771-bb18-35a1464bde6e-kube-api-access-ns9hp\") pod \"maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd\" (UID: \"2896a029-6b06-4771-bb18-35a1464bde6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd" Jul 28 01:44:48.566210 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:48.566004 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2896a029-6b06-4771-bb18-35a1464bde6e-openshift-service-ca\") pod \"maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd\" (UID: \"2896a029-6b06-4771-bb18-35a1464bde6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd" Jul 28 01:44:48.566210 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:44:48.566174 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-1c712754-serving-cert: secret "maas-api-e2e-ait-1c712754-serving-cert" not found Jul 28 01:44:48.566393 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:44:48.566279 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2896a029-6b06-4771-bb18-35a1464bde6e-maas-api-tls podName:2896a029-6b06-4771-bb18-35a1464bde6e nodeName:}" failed. No retries permitted until 2026-07-28 01:44:49.0662563 +0000 UTC m=+1746.500654608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2896a029-6b06-4771-bb18-35a1464bde6e-maas-api-tls") pod "maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd" (UID: "2896a029-6b06-4771-bb18-35a1464bde6e") : secret "maas-api-e2e-ait-1c712754-serving-cert" not found Jul 28 01:44:48.566747 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:48.566726 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2896a029-6b06-4771-bb18-35a1464bde6e-openshift-service-ca\") pod \"maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd\" (UID: \"2896a029-6b06-4771-bb18-35a1464bde6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd" Jul 28 01:44:48.581649 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:48.581622 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns9hp\" (UniqueName: \"kubernetes.io/projected/2896a029-6b06-4771-bb18-35a1464bde6e-kube-api-access-ns9hp\") pod \"maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd\" (UID: \"2896a029-6b06-4771-bb18-35a1464bde6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd" Jul 28 01:44:49.069178 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:49.069145 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2896a029-6b06-4771-bb18-35a1464bde6e-maas-api-tls\") pod \"maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd\" (UID: \"2896a029-6b06-4771-bb18-35a1464bde6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd" Jul 28 01:44:49.069353 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:44:49.069294 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-1c712754-serving-cert: secret "maas-api-e2e-ait-1c712754-serving-cert" not found Jul 28 01:44:49.069395 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:44:49.069366 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2896a029-6b06-4771-bb18-35a1464bde6e-maas-api-tls podName:2896a029-6b06-4771-bb18-35a1464bde6e nodeName:}" failed. No retries permitted until 2026-07-28 01:44:50.06934544 +0000 UTC m=+1747.503743734 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2896a029-6b06-4771-bb18-35a1464bde6e-maas-api-tls") pod "maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd" (UID: "2896a029-6b06-4771-bb18-35a1464bde6e") : secret "maas-api-e2e-ait-1c712754-serving-cert" not found Jul 28 01:44:50.078296 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:50.078255 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2896a029-6b06-4771-bb18-35a1464bde6e-maas-api-tls\") pod \"maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd\" (UID: \"2896a029-6b06-4771-bb18-35a1464bde6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd" Jul 28 01:44:50.080654 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:50.080633 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2896a029-6b06-4771-bb18-35a1464bde6e-maas-api-tls\") pod \"maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd\" (UID: \"2896a029-6b06-4771-bb18-35a1464bde6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd" Jul 28 01:44:50.227270 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:50.227229 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd" Jul 28 01:44:50.357967 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:50.357944 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd"] Jul 28 01:44:50.359551 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:44:50.359526 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2896a029_6b06_4771_bb18_35a1464bde6e.slice/crio-83723367122f06c62a397c4d8917eb68a684cb09eb50490b62c660ceee41895a WatchSource:0}: Error finding container 83723367122f06c62a397c4d8917eb68a684cb09eb50490b62c660ceee41895a: Status 404 returned error can't find the container with id 83723367122f06c62a397c4d8917eb68a684cb09eb50490b62c660ceee41895a Jul 28 01:44:51.340051 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:51.340006 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd" event={"ID":"2896a029-6b06-4771-bb18-35a1464bde6e","Type":"ContainerStarted","Data":"d5283a2e7115364aea2926d43711535e062ee20ec13dd74ae39071155d28cac2"} Jul 28 01:44:51.340457 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:51.340083 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd" event={"ID":"2896a029-6b06-4771-bb18-35a1464bde6e","Type":"ContainerStarted","Data":"83723367122f06c62a397c4d8917eb68a684cb09eb50490b62c660ceee41895a"} Jul 28 01:44:51.340457 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:51.340116 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd" Jul 28 01:44:51.357676 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:51.357624 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd" podStartSLOduration=2.979435045 podStartE2EDuration="3.357610998s" totalImagesPullingTime="378.175953ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:44:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:44:51.356384437 +0000 UTC m=+1748.790782764" watchObservedRunningTime="2026-07-28 01:44:51.357610998 +0000 UTC m=+1748.792009311" Jul 28 01:44:57.349027 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:44:57.348995 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd" Jul 28 01:45:00.128707 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:00.128668 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753385-kzj5q"] Jul 28 01:45:00.132364 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:00.132346 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c712754-29753385-wpf74"] Jul 28 01:45:00.132479 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:00.132459 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753385-kzj5q" Jul 28 01:45:00.135224 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:00.135196 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wjxvt\"" Jul 28 01:45:00.135364 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:00.135353 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c712754-29753385-wpf74" Jul 28 01:45:00.138659 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:00.138630 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753385-kzj5q"] Jul 28 01:45:00.142188 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:00.142162 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c712754-29753385-wpf74"] Jul 28 01:45:00.148578 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:00.148552 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753385-kzj5q" Jul 28 01:45:00.154386 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:00.154366 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c712754-29753385-wpf74" Jul 28 01:45:00.304690 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:00.304663 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753385-kzj5q"] Jul 28 01:45:00.305926 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:45:00.305896 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod869473ef_831f_406d_a325_0959e20d25c2.slice/crio-c0f01332b0a50a9cc5281c37461e8a263efebea8939218d464f4b4ec4e86b74c WatchSource:0}: Error finding container c0f01332b0a50a9cc5281c37461e8a263efebea8939218d464f4b4ec4e86b74c: Status 404 returned error can't find the container with id c0f01332b0a50a9cc5281c37461e8a263efebea8939218d464f4b4ec4e86b74c Jul 28 01:45:00.385275 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:00.385182 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753385-kzj5q" event={"ID":"869473ef-831f-406d-a325-0959e20d25c2","Type":"ContainerStarted","Data":"c0f01332b0a50a9cc5281c37461e8a263efebea8939218d464f4b4ec4e86b74c"} Jul 28 01:45:00.524413 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:00.524383 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c712754-29753385-wpf74"] Jul 28 01:45:00.525403 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:45:00.525378 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod189b6075_1700_450a_96ac_97002be6b82e.slice/crio-a62f5e3bea74e879a490c243307bb6667968363e81b33c5fd96f7af41a646fdd WatchSource:0}: Error finding container a62f5e3bea74e879a490c243307bb6667968363e81b33c5fd96f7af41a646fdd: Status 404 returned error can't find the container with id a62f5e3bea74e879a490c243307bb6667968363e81b33c5fd96f7af41a646fdd Jul 28 01:45:01.397001 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:01.396935 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c712754-29753385-wpf74" event={"ID":"189b6075-1700-450a-96ac-97002be6b82e","Type":"ContainerStarted","Data":"a62f5e3bea74e879a490c243307bb6667968363e81b33c5fd96f7af41a646fdd"} Jul 28 01:45:03.409100 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:03.409058 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753385-kzj5q" event={"ID":"869473ef-831f-406d-a325-0959e20d25c2","Type":"ContainerStarted","Data":"948f8d13077bc6773f2c14c311d6f118c6feef1d33cdb8d9b80add60e8b4773b"} Jul 28 01:45:03.410572 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:03.410545 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c712754-29753385-wpf74" event={"ID":"189b6075-1700-450a-96ac-97002be6b82e","Type":"ContainerStarted","Data":"3c3fdf2bcde14a00a59b5078adb95ed8f87c4ef056da9a09fee0448488bf12f4"} Jul 28 01:45:03.424156 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:03.424108 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753385-kzj5q" podStartSLOduration=1.171882284 podStartE2EDuration="3.424095258s" totalImagesPullingTime="2.252212974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:45:03.42370111 +0000 UTC m=+1760.858099424" watchObservedRunningTime="2026-07-28 01:45:03.424095258 +0000 UTC m=+1760.858493571" Jul 28 01:45:03.438599 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:03.438552 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c712754-29753385-wpf74" podStartSLOduration=1.056591388 podStartE2EDuration="3.438538197s" totalImagesPullingTime="2.381946809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:45:03.437623023 +0000 UTC m=+1760.872021338" watchObservedRunningTime="2026-07-28 01:45:03.438538197 +0000 UTC m=+1760.872936511" Jul 28 01:45:04.416364 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:04.416329 2565 generic.go:356] "Generic (PLEG): container finished" podID="869473ef-831f-406d-a325-0959e20d25c2" containerID="948f8d13077bc6773f2c14c311d6f118c6feef1d33cdb8d9b80add60e8b4773b" exitCode=0 Jul 28 01:45:04.416837 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:04.416415 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753385-kzj5q" event={"ID":"869473ef-831f-406d-a325-0959e20d25c2","Type":"ContainerDied","Data":"948f8d13077bc6773f2c14c311d6f118c6feef1d33cdb8d9b80add60e8b4773b"} Jul 28 01:45:04.417819 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:04.417799 2565 generic.go:356] "Generic (PLEG): container finished" podID="189b6075-1700-450a-96ac-97002be6b82e" containerID="3c3fdf2bcde14a00a59b5078adb95ed8f87c4ef056da9a09fee0448488bf12f4" exitCode=0 Jul 28 01:45:04.417901 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:04.417880 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c712754-29753385-wpf74" event={"ID":"189b6075-1700-450a-96ac-97002be6b82e","Type":"ContainerDied","Data":"3c3fdf2bcde14a00a59b5078adb95ed8f87c4ef056da9a09fee0448488bf12f4"} Jul 28 01:45:05.566302 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:05.566276 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c712754-29753385-wpf74" Jul 28 01:45:05.570174 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:05.570153 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753385-kzj5q" Jul 28 01:45:06.428780 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:06.428743 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753385-kzj5q" Jul 28 01:45:06.428974 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:06.428743 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753385-kzj5q" event={"ID":"869473ef-831f-406d-a325-0959e20d25c2","Type":"ContainerDied","Data":"c0f01332b0a50a9cc5281c37461e8a263efebea8939218d464f4b4ec4e86b74c"} Jul 28 01:45:06.428974 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:06.428862 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0f01332b0a50a9cc5281c37461e8a263efebea8939218d464f4b4ec4e86b74c" Jul 28 01:45:06.430048 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:06.430017 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c712754-29753385-wpf74" Jul 28 01:45:06.430140 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:06.430060 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c712754-29753385-wpf74" event={"ID":"189b6075-1700-450a-96ac-97002be6b82e","Type":"ContainerDied","Data":"a62f5e3bea74e879a490c243307bb6667968363e81b33c5fd96f7af41a646fdd"} Jul 28 01:45:06.430140 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:06.430099 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a62f5e3bea74e879a490c243307bb6667968363e81b33c5fd96f7af41a646fdd" Jul 28 01:45:14.079709 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:14.079670 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c712754-qkcs8"] Jul 28 01:45:14.080255 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:14.080009 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="189b6075-1700-450a-96ac-97002be6b82e" containerName="cleanup" Jul 28 01:45:14.080255 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:14.080020 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="189b6075-1700-450a-96ac-97002be6b82e" containerName="cleanup" Jul 28 01:45:14.080255 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:14.080054 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="869473ef-831f-406d-a325-0959e20d25c2" containerName="cleanup" Jul 28 01:45:14.080255 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:14.080060 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="869473ef-831f-406d-a325-0959e20d25c2" containerName="cleanup" Jul 28 01:45:14.080255 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:14.080117 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="189b6075-1700-450a-96ac-97002be6b82e" containerName="cleanup" Jul 28 01:45:14.080255 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:14.080124 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="869473ef-831f-406d-a325-0959e20d25c2" containerName="cleanup" Jul 28 01:45:14.083249 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:14.083228 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c712754-qkcs8" Jul 28 01:45:14.085826 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:14.085768 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wjxvt\"" Jul 28 01:45:14.088737 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:14.088718 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c712754-qkcs8"] Jul 28 01:45:14.199702 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:14.199661 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34fc56f2-95e3-41e6-bbac-d12ce1c33c77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c712754-qkcs8\" (UID: \"34fc56f2-95e3-41e6-bbac-d12ce1c33c77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c712754-qkcs8" Jul 28 01:45:14.300350 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:14.300319 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34fc56f2-95e3-41e6-bbac-d12ce1c33c77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c712754-qkcs8\" (UID: \"34fc56f2-95e3-41e6-bbac-d12ce1c33c77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c712754-qkcs8" Jul 28 01:45:14.300963 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:14.300943 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34fc56f2-95e3-41e6-bbac-d12ce1c33c77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c712754-qkcs8\" (UID: \"34fc56f2-95e3-41e6-bbac-d12ce1c33c77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c712754-qkcs8" Jul 28 01:45:14.395220 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:14.395128 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c712754-qkcs8" Jul 28 01:45:14.523925 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:14.523901 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c712754-qkcs8"] Jul 28 01:45:14.525818 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:45:14.525794 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34fc56f2_95e3_41e6_bbac_d12ce1c33c77.slice/crio-46d74e0bd49b9820e5bd0911ed71c210efb71c45e34f4cf8618b45ab78a2c8e9 WatchSource:0}: Error finding container 46d74e0bd49b9820e5bd0911ed71c210efb71c45e34f4cf8618b45ab78a2c8e9: Status 404 returned error can't find the container with id 46d74e0bd49b9820e5bd0911ed71c210efb71c45e34f4cf8618b45ab78a2c8e9 Jul 28 01:45:15.476128 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:15.476089 2565 generic.go:356] "Generic (PLEG): container finished" podID="34fc56f2-95e3-41e6-bbac-d12ce1c33c77" containerID="42603dd8a049e0ff7d278dbb39f21d570aebaf236c648b593498991a31d78ba8" exitCode=0 Jul 28 01:45:15.476504 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:15.476146 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c712754-qkcs8" event={"ID":"34fc56f2-95e3-41e6-bbac-d12ce1c33c77","Type":"ContainerDied","Data":"42603dd8a049e0ff7d278dbb39f21d570aebaf236c648b593498991a31d78ba8"} Jul 28 01:45:15.476504 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:15.476190 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c712754-qkcs8" event={"ID":"34fc56f2-95e3-41e6-bbac-d12ce1c33c77","Type":"ContainerStarted","Data":"46d74e0bd49b9820e5bd0911ed71c210efb71c45e34f4cf8618b45ab78a2c8e9"} Jul 28 01:45:16.612791 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:16.612765 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c712754-qkcs8" Jul 28 01:45:16.723055 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:16.723000 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34fc56f2-95e3-41e6-bbac-d12ce1c33c77-maas-api-service-ca\") pod \"34fc56f2-95e3-41e6-bbac-d12ce1c33c77\" (UID: \"34fc56f2-95e3-41e6-bbac-d12ce1c33c77\") " Jul 28 01:45:16.723345 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:16.723318 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34fc56f2-95e3-41e6-bbac-d12ce1c33c77-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "34fc56f2-95e3-41e6-bbac-d12ce1c33c77" (UID: "34fc56f2-95e3-41e6-bbac-d12ce1c33c77"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:45:16.824592 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:16.824501 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34fc56f2-95e3-41e6-bbac-d12ce1c33c77-maas-api-service-ca\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:45:17.487257 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:17.487227 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c712754-qkcs8" Jul 28 01:45:17.487435 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:17.487251 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c712754-qkcs8" event={"ID":"34fc56f2-95e3-41e6-bbac-d12ce1c33c77","Type":"ContainerDied","Data":"46d74e0bd49b9820e5bd0911ed71c210efb71c45e34f4cf8618b45ab78a2c8e9"} Jul 28 01:45:17.487435 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:17.487307 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46d74e0bd49b9820e5bd0911ed71c210efb71c45e34f4cf8618b45ab78a2c8e9" Jul 28 01:45:24.590100 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:24.590057 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd"] Jul 28 01:45:24.590505 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:24.590327 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd" podUID="2896a029-6b06-4771-bb18-35a1464bde6e" containerName="maas-api" containerID="cri-o://d5283a2e7115364aea2926d43711535e062ee20ec13dd74ae39071155d28cac2" gracePeriod=30 Jul 28 01:45:24.616525 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:24.616497 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c712754-29753385-wpf74"] Jul 28 01:45:24.619203 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:24.619176 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c712754-29753385-wpf74"] Jul 28 01:45:24.832359 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:24.832334 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd" Jul 28 01:45:24.893859 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:24.893768 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2896a029-6b06-4771-bb18-35a1464bde6e-maas-api-tls\") pod \"2896a029-6b06-4771-bb18-35a1464bde6e\" (UID: \"2896a029-6b06-4771-bb18-35a1464bde6e\") " Jul 28 01:45:24.893859 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:24.893852 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2896a029-6b06-4771-bb18-35a1464bde6e-openshift-service-ca\") pod \"2896a029-6b06-4771-bb18-35a1464bde6e\" (UID: \"2896a029-6b06-4771-bb18-35a1464bde6e\") " Jul 28 01:45:24.894092 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:24.893891 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ns9hp\" (UniqueName: \"kubernetes.io/projected/2896a029-6b06-4771-bb18-35a1464bde6e-kube-api-access-ns9hp\") pod \"2896a029-6b06-4771-bb18-35a1464bde6e\" (UID: \"2896a029-6b06-4771-bb18-35a1464bde6e\") " Jul 28 01:45:24.894320 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:24.894294 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2896a029-6b06-4771-bb18-35a1464bde6e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2896a029-6b06-4771-bb18-35a1464bde6e" (UID: "2896a029-6b06-4771-bb18-35a1464bde6e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:45:24.895843 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:24.895826 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2896a029-6b06-4771-bb18-35a1464bde6e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2896a029-6b06-4771-bb18-35a1464bde6e" (UID: "2896a029-6b06-4771-bb18-35a1464bde6e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:45:24.896061 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:24.896046 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2896a029-6b06-4771-bb18-35a1464bde6e-kube-api-access-ns9hp" (OuterVolumeSpecName: "kube-api-access-ns9hp") pod "2896a029-6b06-4771-bb18-35a1464bde6e" (UID: "2896a029-6b06-4771-bb18-35a1464bde6e"). InnerVolumeSpecName "kube-api-access-ns9hp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:45:24.995596 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:24.995553 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2896a029-6b06-4771-bb18-35a1464bde6e-maas-api-tls\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:45:24.995596 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:24.995594 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2896a029-6b06-4771-bb18-35a1464bde6e-openshift-service-ca\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:45:24.995596 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:24.995604 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns9hp\" (UniqueName: \"kubernetes.io/projected/2896a029-6b06-4771-bb18-35a1464bde6e-kube-api-access-ns9hp\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:45:25.101000 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:25.100968 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="189b6075-1700-450a-96ac-97002be6b82e" path="/var/lib/kubelet/pods/189b6075-1700-450a-96ac-97002be6b82e/volumes" Jul 28 01:45:25.529491 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:25.529452 2565 generic.go:356] "Generic (PLEG): container finished" podID="2896a029-6b06-4771-bb18-35a1464bde6e" containerID="d5283a2e7115364aea2926d43711535e062ee20ec13dd74ae39071155d28cac2" exitCode=0 Jul 28 01:45:25.529666 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:25.529516 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd" event={"ID":"2896a029-6b06-4771-bb18-35a1464bde6e","Type":"ContainerDied","Data":"d5283a2e7115364aea2926d43711535e062ee20ec13dd74ae39071155d28cac2"} Jul 28 01:45:25.529666 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:25.529532 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd" Jul 28 01:45:25.529666 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:25.529571 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd" event={"ID":"2896a029-6b06-4771-bb18-35a1464bde6e","Type":"ContainerDied","Data":"83723367122f06c62a397c4d8917eb68a684cb09eb50490b62c660ceee41895a"} Jul 28 01:45:25.529666 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:25.529596 2565 scope.go:117] "RemoveContainer" containerID="d5283a2e7115364aea2926d43711535e062ee20ec13dd74ae39071155d28cac2" Jul 28 01:45:25.538096 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:25.538077 2565 scope.go:117] "RemoveContainer" containerID="d5283a2e7115364aea2926d43711535e062ee20ec13dd74ae39071155d28cac2" Jul 28 01:45:25.538395 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:45:25.538371 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5283a2e7115364aea2926d43711535e062ee20ec13dd74ae39071155d28cac2\": container with ID starting with d5283a2e7115364aea2926d43711535e062ee20ec13dd74ae39071155d28cac2 not found: ID does not exist" containerID="d5283a2e7115364aea2926d43711535e062ee20ec13dd74ae39071155d28cac2" Jul 28 01:45:25.538470 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:25.538407 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5283a2e7115364aea2926d43711535e062ee20ec13dd74ae39071155d28cac2"} err="failed to get container status \"d5283a2e7115364aea2926d43711535e062ee20ec13dd74ae39071155d28cac2\": rpc error: code = NotFound desc = could not find container \"d5283a2e7115364aea2926d43711535e062ee20ec13dd74ae39071155d28cac2\": container with ID starting with d5283a2e7115364aea2926d43711535e062ee20ec13dd74ae39071155d28cac2 not found: ID does not exist" Jul 28 01:45:25.546278 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:25.546256 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd"] Jul 28 01:45:25.549835 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:25.549811 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c712754-6f946bdd9d-9h8kd"] Jul 28 01:45:27.100910 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:27.100874 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2896a029-6b06-4771-bb18-35a1464bde6e" path="/var/lib/kubelet/pods/2896a029-6b06-4771-bb18-35a1464bde6e/volumes" Jul 28 01:45:29.164287 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:29.164184 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c712754-qkcs8"] Jul 28 01:45:29.166831 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:29.166808 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c712754-qkcs8"] Jul 28 01:45:31.101887 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:31.101852 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34fc56f2-95e3-41e6-bbac-d12ce1c33c77" path="/var/lib/kubelet/pods/34fc56f2-95e3-41e6-bbac-d12ce1c33c77/volumes" Jul 28 01:45:33.658731 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:33.658693 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn"] Jul 28 01:45:33.659139 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:33.659113 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34fc56f2-95e3-41e6-bbac-d12ce1c33c77" containerName="revoke-keys" Jul 28 01:45:33.659139 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:33.659130 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="34fc56f2-95e3-41e6-bbac-d12ce1c33c77" containerName="revoke-keys" Jul 28 01:45:33.659262 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:33.659148 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2896a029-6b06-4771-bb18-35a1464bde6e" containerName="maas-api" Jul 28 01:45:33.659262 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:33.659153 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2896a029-6b06-4771-bb18-35a1464bde6e" containerName="maas-api" Jul 28 01:45:33.659262 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:33.659207 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="34fc56f2-95e3-41e6-bbac-d12ce1c33c77" containerName="revoke-keys" Jul 28 01:45:33.659262 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:33.659217 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2896a029-6b06-4771-bb18-35a1464bde6e" containerName="maas-api" Jul 28 01:45:33.664563 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:33.664541 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn" Jul 28 01:45:33.667507 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:33.667485 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-5442bcc0-serving-cert\"" Jul 28 01:45:33.667611 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:33.667523 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-62vvq\"" Jul 28 01:45:33.667717 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:33.667698 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 01:45:33.668542 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:33.668523 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 28 01:45:33.672377 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:33.672355 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn"] Jul 28 01:45:33.773226 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:33.773189 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95ed4c7b-e7fc-4552-af0d-5aabd5a26471-openshift-service-ca\") pod \"maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn\" (UID: \"95ed4c7b-e7fc-4552-af0d-5aabd5a26471\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn" Jul 28 01:45:33.773411 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:33.773271 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95ed4c7b-e7fc-4552-af0d-5aabd5a26471-maas-api-tls\") pod \"maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn\" (UID: \"95ed4c7b-e7fc-4552-af0d-5aabd5a26471\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn" Jul 28 01:45:33.773411 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:33.773319 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q2lg\" (UniqueName: \"kubernetes.io/projected/95ed4c7b-e7fc-4552-af0d-5aabd5a26471-kube-api-access-6q2lg\") pod \"maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn\" (UID: \"95ed4c7b-e7fc-4552-af0d-5aabd5a26471\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn" Jul 28 01:45:33.874329 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:33.874288 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95ed4c7b-e7fc-4552-af0d-5aabd5a26471-openshift-service-ca\") pod \"maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn\" (UID: \"95ed4c7b-e7fc-4552-af0d-5aabd5a26471\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn" Jul 28 01:45:33.874524 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:33.874351 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95ed4c7b-e7fc-4552-af0d-5aabd5a26471-maas-api-tls\") pod \"maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn\" (UID: \"95ed4c7b-e7fc-4552-af0d-5aabd5a26471\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn" Jul 28 01:45:33.874524 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:33.874384 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q2lg\" (UniqueName: \"kubernetes.io/projected/95ed4c7b-e7fc-4552-af0d-5aabd5a26471-kube-api-access-6q2lg\") pod \"maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn\" (UID: \"95ed4c7b-e7fc-4552-af0d-5aabd5a26471\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn" Jul 28 01:45:33.874624 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:45:33.874514 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-5442bcc0-serving-cert: secret "maas-api-e2e-derive-5442bcc0-serving-cert" not found Jul 28 01:45:33.874624 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:45:33.874605 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95ed4c7b-e7fc-4552-af0d-5aabd5a26471-maas-api-tls podName:95ed4c7b-e7fc-4552-af0d-5aabd5a26471 nodeName:}" failed. No retries permitted until 2026-07-28 01:45:34.374583584 +0000 UTC m=+1791.808981895 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/95ed4c7b-e7fc-4552-af0d-5aabd5a26471-maas-api-tls") pod "maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn" (UID: "95ed4c7b-e7fc-4552-af0d-5aabd5a26471") : secret "maas-api-e2e-derive-5442bcc0-serving-cert" not found Jul 28 01:45:33.875093 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:33.875074 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95ed4c7b-e7fc-4552-af0d-5aabd5a26471-openshift-service-ca\") pod \"maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn\" (UID: \"95ed4c7b-e7fc-4552-af0d-5aabd5a26471\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn" Jul 28 01:45:33.886050 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:33.886010 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q2lg\" (UniqueName: \"kubernetes.io/projected/95ed4c7b-e7fc-4552-af0d-5aabd5a26471-kube-api-access-6q2lg\") pod \"maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn\" (UID: \"95ed4c7b-e7fc-4552-af0d-5aabd5a26471\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn" Jul 28 01:45:34.378801 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:34.378762 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95ed4c7b-e7fc-4552-af0d-5aabd5a26471-maas-api-tls\") pod \"maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn\" (UID: \"95ed4c7b-e7fc-4552-af0d-5aabd5a26471\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn" Jul 28 01:45:34.381689 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:34.381663 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95ed4c7b-e7fc-4552-af0d-5aabd5a26471-maas-api-tls\") pod \"maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn\" (UID: \"95ed4c7b-e7fc-4552-af0d-5aabd5a26471\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn" Jul 28 01:45:34.577196 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:34.577160 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn" Jul 28 01:45:34.711588 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:34.711560 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn"] Jul 28 01:45:34.713417 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:45:34.713390 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95ed4c7b_e7fc_4552_af0d_5aabd5a26471.slice/crio-7335162af1643dfecafab99fa42765dbbabd9fe398578b7f33c3d398f9fce16d WatchSource:0}: Error finding container 7335162af1643dfecafab99fa42765dbbabd9fe398578b7f33c3d398f9fce16d: Status 404 returned error can't find the container with id 7335162af1643dfecafab99fa42765dbbabd9fe398578b7f33c3d398f9fce16d Jul 28 01:45:35.581683 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:35.581642 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn" event={"ID":"95ed4c7b-e7fc-4552-af0d-5aabd5a26471","Type":"ContainerStarted","Data":"7b5a54b3fea5d0e4d0a5cf81b58a6e04c07b422f5066587f449df1e7bae6b340"} Jul 28 01:45:35.581851 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:35.581706 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn" event={"ID":"95ed4c7b-e7fc-4552-af0d-5aabd5a26471","Type":"ContainerStarted","Data":"7335162af1643dfecafab99fa42765dbbabd9fe398578b7f33c3d398f9fce16d"} Jul 28 01:45:35.581851 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:35.581785 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn" Jul 28 01:45:35.598238 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:35.598187 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn" podStartSLOduration=2.256923331 podStartE2EDuration="2.598171795s" totalImagesPullingTime="341.248464ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:45:35.597507699 +0000 UTC m=+1793.031906013" watchObservedRunningTime="2026-07-28 01:45:35.598171795 +0000 UTC m=+1793.032570109" Jul 28 01:45:41.592996 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:41.592966 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn" Jul 28 01:45:58.504442 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:58.504403 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5442bcc0-b4hwd"] Jul 28 01:45:58.506831 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:58.506814 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5442bcc0-b4hwd" Jul 28 01:45:58.509504 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:58.509482 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wjxvt\"" Jul 28 01:45:58.513622 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:58.513594 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5442bcc0-b4hwd"] Jul 28 01:45:58.693189 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:58.693155 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1125cd2-e166-496b-882e-39bf042699d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-5442bcc0-b4hwd\" (UID: \"f1125cd2-e166-496b-882e-39bf042699d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5442bcc0-b4hwd" Jul 28 01:45:58.794623 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:58.794538 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1125cd2-e166-496b-882e-39bf042699d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-5442bcc0-b4hwd\" (UID: \"f1125cd2-e166-496b-882e-39bf042699d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5442bcc0-b4hwd" Jul 28 01:45:58.795193 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:58.795171 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1125cd2-e166-496b-882e-39bf042699d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-5442bcc0-b4hwd\" (UID: \"f1125cd2-e166-496b-882e-39bf042699d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5442bcc0-b4hwd" Jul 28 01:45:58.818545 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:58.818524 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5442bcc0-b4hwd" Jul 28 01:45:58.946608 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:58.946576 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5442bcc0-b4hwd"] Jul 28 01:45:58.948870 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:45:58.948843 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1125cd2_e166_496b_882e_39bf042699d4.slice/crio-3f1176cb09283200b2c01c9ee54b2e53a1402d54c36197cdbca426952f5a9d68 WatchSource:0}: Error finding container 3f1176cb09283200b2c01c9ee54b2e53a1402d54c36197cdbca426952f5a9d68: Status 404 returned error can't find the container with id 3f1176cb09283200b2c01c9ee54b2e53a1402d54c36197cdbca426952f5a9d68 Jul 28 01:45:59.700580 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:59.700548 2565 generic.go:356] "Generic (PLEG): container finished" podID="f1125cd2-e166-496b-882e-39bf042699d4" containerID="42fa9a6d6100eaee5ede00e43be2331a8c309051afea4bcf34ca8ddc87f91182" exitCode=0 Jul 28 01:45:59.700947 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:59.700586 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5442bcc0-b4hwd" event={"ID":"f1125cd2-e166-496b-882e-39bf042699d4","Type":"ContainerDied","Data":"42fa9a6d6100eaee5ede00e43be2331a8c309051afea4bcf34ca8ddc87f91182"} Jul 28 01:45:59.700947 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:45:59.700626 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5442bcc0-b4hwd" event={"ID":"f1125cd2-e166-496b-882e-39bf042699d4","Type":"ContainerStarted","Data":"3f1176cb09283200b2c01c9ee54b2e53a1402d54c36197cdbca426952f5a9d68"} Jul 28 01:46:00.833813 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:46:00.833786 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5442bcc0-b4hwd" Jul 28 01:46:01.014867 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:46:01.014780 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1125cd2-e166-496b-882e-39bf042699d4-maas-api-service-ca\") pod \"f1125cd2-e166-496b-882e-39bf042699d4\" (UID: \"f1125cd2-e166-496b-882e-39bf042699d4\") " Jul 28 01:46:01.015183 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:46:01.015155 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1125cd2-e166-496b-882e-39bf042699d4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f1125cd2-e166-496b-882e-39bf042699d4" (UID: "f1125cd2-e166-496b-882e-39bf042699d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:46:01.115335 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:46:01.115310 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1125cd2-e166-496b-882e-39bf042699d4-maas-api-service-ca\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:46:01.715864 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:46:01.715825 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5442bcc0-b4hwd" Jul 28 01:46:01.716075 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:46:01.715861 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5442bcc0-b4hwd" event={"ID":"f1125cd2-e166-496b-882e-39bf042699d4","Type":"ContainerDied","Data":"3f1176cb09283200b2c01c9ee54b2e53a1402d54c36197cdbca426952f5a9d68"} Jul 28 01:46:01.716075 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:46:01.715913 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f1176cb09283200b2c01c9ee54b2e53a1402d54c36197cdbca426952f5a9d68" Jul 28 01:46:08.917677 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:46:08.917638 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn"] Jul 28 01:46:08.918128 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:46:08.917897 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn" podUID="95ed4c7b-e7fc-4552-af0d-5aabd5a26471" containerName="maas-api" containerID="cri-o://7b5a54b3fea5d0e4d0a5cf81b58a6e04c07b422f5066587f449df1e7bae6b340" gracePeriod=30 Jul 28 01:46:09.171808 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:46:09.171752 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn" Jul 28 01:46:09.271463 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:46:09.271431 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6q2lg\" (UniqueName: \"kubernetes.io/projected/95ed4c7b-e7fc-4552-af0d-5aabd5a26471-kube-api-access-6q2lg\") pod \"95ed4c7b-e7fc-4552-af0d-5aabd5a26471\" (UID: \"95ed4c7b-e7fc-4552-af0d-5aabd5a26471\") " Jul 28 01:46:09.271653 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:46:09.271473 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95ed4c7b-e7fc-4552-af0d-5aabd5a26471-openshift-service-ca\") pod \"95ed4c7b-e7fc-4552-af0d-5aabd5a26471\" (UID: \"95ed4c7b-e7fc-4552-af0d-5aabd5a26471\") " Jul 28 01:46:09.271653 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:46:09.271541 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95ed4c7b-e7fc-4552-af0d-5aabd5a26471-maas-api-tls\") pod \"95ed4c7b-e7fc-4552-af0d-5aabd5a26471\" (UID: \"95ed4c7b-e7fc-4552-af0d-5aabd5a26471\") " Jul 28 01:46:09.271887 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:46:09.271867 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95ed4c7b-e7fc-4552-af0d-5aabd5a26471-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "95ed4c7b-e7fc-4552-af0d-5aabd5a26471" (UID: "95ed4c7b-e7fc-4552-af0d-5aabd5a26471"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:46:09.273573 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:46:09.273549 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95ed4c7b-e7fc-4552-af0d-5aabd5a26471-kube-api-access-6q2lg" (OuterVolumeSpecName: "kube-api-access-6q2lg") pod "95ed4c7b-e7fc-4552-af0d-5aabd5a26471" (UID: "95ed4c7b-e7fc-4552-af0d-5aabd5a26471"). InnerVolumeSpecName "kube-api-access-6q2lg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:46:09.273670 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:46:09.273590 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95ed4c7b-e7fc-4552-af0d-5aabd5a26471-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "95ed4c7b-e7fc-4552-af0d-5aabd5a26471" (UID: "95ed4c7b-e7fc-4552-af0d-5aabd5a26471"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:46:09.372135 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:46:09.372099 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q2lg\" (UniqueName: \"kubernetes.io/projected/95ed4c7b-e7fc-4552-af0d-5aabd5a26471-kube-api-access-6q2lg\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:46:09.372135 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:46:09.372127 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95ed4c7b-e7fc-4552-af0d-5aabd5a26471-openshift-service-ca\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:46:09.372135 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:46:09.372140 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95ed4c7b-e7fc-4552-af0d-5aabd5a26471-maas-api-tls\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:46:09.760280 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:46:09.760246 2565 generic.go:356] "Generic (PLEG): container finished" podID="95ed4c7b-e7fc-4552-af0d-5aabd5a26471" containerID="7b5a54b3fea5d0e4d0a5cf81b58a6e04c07b422f5066587f449df1e7bae6b340" exitCode=0 Jul 28 01:46:09.760456 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:46:09.760298 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn" Jul 28 01:46:09.760456 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:46:09.760337 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn" event={"ID":"95ed4c7b-e7fc-4552-af0d-5aabd5a26471","Type":"ContainerDied","Data":"7b5a54b3fea5d0e4d0a5cf81b58a6e04c07b422f5066587f449df1e7bae6b340"} Jul 28 01:46:09.760456 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:46:09.760396 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn" event={"ID":"95ed4c7b-e7fc-4552-af0d-5aabd5a26471","Type":"ContainerDied","Data":"7335162af1643dfecafab99fa42765dbbabd9fe398578b7f33c3d398f9fce16d"} Jul 28 01:46:09.760456 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:46:09.760421 2565 scope.go:117] "RemoveContainer" containerID="7b5a54b3fea5d0e4d0a5cf81b58a6e04c07b422f5066587f449df1e7bae6b340" Jul 28 01:46:09.769443 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:46:09.769422 2565 scope.go:117] "RemoveContainer" containerID="7b5a54b3fea5d0e4d0a5cf81b58a6e04c07b422f5066587f449df1e7bae6b340" Jul 28 01:46:09.769693 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:46:09.769676 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b5a54b3fea5d0e4d0a5cf81b58a6e04c07b422f5066587f449df1e7bae6b340\": container with ID starting with 7b5a54b3fea5d0e4d0a5cf81b58a6e04c07b422f5066587f449df1e7bae6b340 not found: ID does not exist" containerID="7b5a54b3fea5d0e4d0a5cf81b58a6e04c07b422f5066587f449df1e7bae6b340" Jul 28 01:46:09.769761 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:46:09.769702 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b5a54b3fea5d0e4d0a5cf81b58a6e04c07b422f5066587f449df1e7bae6b340"} err="failed to get container status \"7b5a54b3fea5d0e4d0a5cf81b58a6e04c07b422f5066587f449df1e7bae6b340\": rpc error: code = NotFound desc = could not find container \"7b5a54b3fea5d0e4d0a5cf81b58a6e04c07b422f5066587f449df1e7bae6b340\": container with ID starting with 7b5a54b3fea5d0e4d0a5cf81b58a6e04c07b422f5066587f449df1e7bae6b340 not found: ID does not exist" Jul 28 01:46:09.781834 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:46:09.781810 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn"] Jul 28 01:46:09.783847 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:46:09.783827 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5442bcc0-69864b675d-pzpdn"] Jul 28 01:46:11.104331 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:46:11.104286 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95ed4c7b-e7fc-4552-af0d-5aabd5a26471" path="/var/lib/kubelet/pods/95ed4c7b-e7fc-4552-af0d-5aabd5a26471/volumes" Jul 28 01:46:13.578186 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:46:13.578150 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5442bcc0-b4hwd"] Jul 28 01:46:13.580314 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:46:13.580284 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5442bcc0-b4hwd"] Jul 28 01:46:15.101273 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:46:15.101240 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1125cd2-e166-496b-882e-39bf042699d4" path="/var/lib/kubelet/pods/f1125cd2-e166-496b-882e-39bf042699d4/volumes" Jul 28 01:48:02.161986 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:02.161949 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn"] Jul 28 01:48:02.162611 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:02.162417 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1125cd2-e166-496b-882e-39bf042699d4" containerName="revoke-keys" Jul 28 01:48:02.162611 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:02.162458 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1125cd2-e166-496b-882e-39bf042699d4" containerName="revoke-keys" Jul 28 01:48:02.162611 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:02.162478 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95ed4c7b-e7fc-4552-af0d-5aabd5a26471" containerName="maas-api" Jul 28 01:48:02.162611 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:02.162490 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="95ed4c7b-e7fc-4552-af0d-5aabd5a26471" containerName="maas-api" Jul 28 01:48:02.162611 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:02.162583 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1125cd2-e166-496b-882e-39bf042699d4" containerName="revoke-keys" Jul 28 01:48:02.162611 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:02.162596 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="95ed4c7b-e7fc-4552-af0d-5aabd5a26471" containerName="maas-api" Jul 28 01:48:02.169790 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:02.169757 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn" Jul 28 01:48:02.173326 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:02.173296 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-43a6ae00-serving-cert\"" Jul 28 01:48:02.173469 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:02.173386 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 01:48:02.173469 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:02.173439 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-62vvq\"" Jul 28 01:48:02.173906 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:02.173883 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 28 01:48:02.175205 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:02.175183 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn"] Jul 28 01:48:02.265371 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:02.265336 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7tv5\" (UniqueName: \"kubernetes.io/projected/c6037fca-f880-4656-86be-a3bd0c003dad-kube-api-access-r7tv5\") pod \"maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn\" (UID: \"c6037fca-f880-4656-86be-a3bd0c003dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn" Jul 28 01:48:02.265529 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:02.265395 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6037fca-f880-4656-86be-a3bd0c003dad-maas-api-tls\") pod \"maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn\" (UID: \"c6037fca-f880-4656-86be-a3bd0c003dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn" Jul 28 01:48:02.265529 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:02.265483 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6037fca-f880-4656-86be-a3bd0c003dad-openshift-service-ca\") pod \"maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn\" (UID: \"c6037fca-f880-4656-86be-a3bd0c003dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn" Jul 28 01:48:02.366545 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:02.366508 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6037fca-f880-4656-86be-a3bd0c003dad-maas-api-tls\") pod \"maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn\" (UID: \"c6037fca-f880-4656-86be-a3bd0c003dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn" Jul 28 01:48:02.366735 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:02.366551 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6037fca-f880-4656-86be-a3bd0c003dad-openshift-service-ca\") pod \"maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn\" (UID: \"c6037fca-f880-4656-86be-a3bd0c003dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn" Jul 28 01:48:02.366735 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:02.366616 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7tv5\" (UniqueName: \"kubernetes.io/projected/c6037fca-f880-4656-86be-a3bd0c003dad-kube-api-access-r7tv5\") pod \"maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn\" (UID: \"c6037fca-f880-4656-86be-a3bd0c003dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn" Jul 28 01:48:02.367266 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:02.367242 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6037fca-f880-4656-86be-a3bd0c003dad-openshift-service-ca\") pod \"maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn\" (UID: \"c6037fca-f880-4656-86be-a3bd0c003dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn" Jul 28 01:48:02.368920 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:02.368898 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6037fca-f880-4656-86be-a3bd0c003dad-maas-api-tls\") pod \"maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn\" (UID: \"c6037fca-f880-4656-86be-a3bd0c003dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn" Jul 28 01:48:02.376356 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:02.376333 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7tv5\" (UniqueName: \"kubernetes.io/projected/c6037fca-f880-4656-86be-a3bd0c003dad-kube-api-access-r7tv5\") pod \"maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn\" (UID: \"c6037fca-f880-4656-86be-a3bd0c003dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn" Jul 28 01:48:02.480866 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:02.480833 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn" Jul 28 01:48:02.619510 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:02.619485 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn"] Jul 28 01:48:02.621001 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:48:02.620976 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6037fca_f880_4656_86be_a3bd0c003dad.slice/crio-af93477de6edbe8da04dbd279d378b076d63cd5b64baf467b4c71d3f6339a3eb WatchSource:0}: Error finding container af93477de6edbe8da04dbd279d378b076d63cd5b64baf467b4c71d3f6339a3eb: Status 404 returned error can't find the container with id af93477de6edbe8da04dbd279d378b076d63cd5b64baf467b4c71d3f6339a3eb Jul 28 01:48:03.322662 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:03.322561 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn" event={"ID":"c6037fca-f880-4656-86be-a3bd0c003dad","Type":"ContainerStarted","Data":"dcb40b6e610f19d5a48296dafb1829ca6cd19dfeb82c0ad722b6ec8f6a075a47"} Jul 28 01:48:03.322662 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:03.322624 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn" event={"ID":"c6037fca-f880-4656-86be-a3bd0c003dad","Type":"ContainerStarted","Data":"af93477de6edbe8da04dbd279d378b076d63cd5b64baf467b4c71d3f6339a3eb"} Jul 28 01:48:03.322662 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:03.322650 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn" Jul 28 01:48:03.340757 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:03.340701 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn" podStartSLOduration=0.94173356 podStartE2EDuration="1.340684853s" totalImagesPullingTime="398.951293ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:48:03.33910996 +0000 UTC m=+1940.773508274" watchObservedRunningTime="2026-07-28 01:48:03.340684853 +0000 UTC m=+1940.775083158" Jul 28 01:48:09.333182 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:09.333152 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn" Jul 28 01:48:13.690352 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:13.690314 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-43a6ae00-xrrnm"] Jul 28 01:48:13.700792 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:13.700754 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-43a6ae00-xrrnm"] Jul 28 01:48:13.700792 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:13.700790 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-43a6ae00-xrrnm" Jul 28 01:48:13.703665 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:13.703638 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wjxvt\"" Jul 28 01:48:13.770345 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:13.770308 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99253286-389e-46ca-b331-8ce6bb5ba28a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-43a6ae00-xrrnm\" (UID: \"99253286-389e-46ca-b331-8ce6bb5ba28a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-43a6ae00-xrrnm" Jul 28 01:48:13.871249 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:13.871208 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99253286-389e-46ca-b331-8ce6bb5ba28a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-43a6ae00-xrrnm\" (UID: \"99253286-389e-46ca-b331-8ce6bb5ba28a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-43a6ae00-xrrnm" Jul 28 01:48:13.871908 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:13.871884 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99253286-389e-46ca-b331-8ce6bb5ba28a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-43a6ae00-xrrnm\" (UID: \"99253286-389e-46ca-b331-8ce6bb5ba28a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-43a6ae00-xrrnm" Jul 28 01:48:14.011590 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:14.011474 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-43a6ae00-xrrnm" Jul 28 01:48:14.149806 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:14.149773 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-43a6ae00-xrrnm"] Jul 28 01:48:14.150921 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:48:14.150891 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99253286_389e_46ca_b331_8ce6bb5ba28a.slice/crio-13fa589d973e98ad6c8b1a5c754461b46f0421a61fdb5d3606ca7499b24843db WatchSource:0}: Error finding container 13fa589d973e98ad6c8b1a5c754461b46f0421a61fdb5d3606ca7499b24843db: Status 404 returned error can't find the container with id 13fa589d973e98ad6c8b1a5c754461b46f0421a61fdb5d3606ca7499b24843db Jul 28 01:48:14.384951 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:14.384917 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-43a6ae00-xrrnm" event={"ID":"99253286-389e-46ca-b331-8ce6bb5ba28a","Type":"ContainerStarted","Data":"13fa589d973e98ad6c8b1a5c754461b46f0421a61fdb5d3606ca7499b24843db"} Jul 28 01:48:15.393248 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:15.393215 2565 generic.go:356] "Generic (PLEG): container finished" podID="99253286-389e-46ca-b331-8ce6bb5ba28a" containerID="a2872a91b13b5969574b8539350362e9c77c8d38051b4e965828dff087a1f5ef" exitCode=0 Jul 28 01:48:15.393664 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:15.393312 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-43a6ae00-xrrnm" event={"ID":"99253286-389e-46ca-b331-8ce6bb5ba28a","Type":"ContainerDied","Data":"a2872a91b13b5969574b8539350362e9c77c8d38051b4e965828dff087a1f5ef"} Jul 28 01:48:16.526630 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:16.526603 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-43a6ae00-xrrnm" Jul 28 01:48:16.595985 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:16.595944 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99253286-389e-46ca-b331-8ce6bb5ba28a-maas-api-service-ca\") pod \"99253286-389e-46ca-b331-8ce6bb5ba28a\" (UID: \"99253286-389e-46ca-b331-8ce6bb5ba28a\") " Jul 28 01:48:16.596390 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:16.596366 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99253286-389e-46ca-b331-8ce6bb5ba28a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "99253286-389e-46ca-b331-8ce6bb5ba28a" (UID: "99253286-389e-46ca-b331-8ce6bb5ba28a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:48:16.697382 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:16.697348 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99253286-389e-46ca-b331-8ce6bb5ba28a-maas-api-service-ca\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:48:17.404606 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:17.404514 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-43a6ae00-xrrnm" Jul 28 01:48:17.404606 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:17.404554 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-43a6ae00-xrrnm" event={"ID":"99253286-389e-46ca-b331-8ce6bb5ba28a","Type":"ContainerDied","Data":"13fa589d973e98ad6c8b1a5c754461b46f0421a61fdb5d3606ca7499b24843db"} Jul 28 01:48:17.404812 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:17.404612 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13fa589d973e98ad6c8b1a5c754461b46f0421a61fdb5d3606ca7499b24843db" Jul 28 01:48:24.125594 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:24.125552 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn"] Jul 28 01:48:24.128251 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:24.125806 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn" podUID="c6037fca-f880-4656-86be-a3bd0c003dad" containerName="maas-api" containerID="cri-o://dcb40b6e610f19d5a48296dafb1829ca6cd19dfeb82c0ad722b6ec8f6a075a47" gracePeriod=30 Jul 28 01:48:24.377322 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:24.377254 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn" Jul 28 01:48:24.439588 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:24.439552 2565 generic.go:356] "Generic (PLEG): container finished" podID="c6037fca-f880-4656-86be-a3bd0c003dad" containerID="dcb40b6e610f19d5a48296dafb1829ca6cd19dfeb82c0ad722b6ec8f6a075a47" exitCode=0 Jul 28 01:48:24.439755 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:24.439595 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn" Jul 28 01:48:24.439755 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:24.439634 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn" event={"ID":"c6037fca-f880-4656-86be-a3bd0c003dad","Type":"ContainerDied","Data":"dcb40b6e610f19d5a48296dafb1829ca6cd19dfeb82c0ad722b6ec8f6a075a47"} Jul 28 01:48:24.439755 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:24.439687 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn" event={"ID":"c6037fca-f880-4656-86be-a3bd0c003dad","Type":"ContainerDied","Data":"af93477de6edbe8da04dbd279d378b076d63cd5b64baf467b4c71d3f6339a3eb"} Jul 28 01:48:24.439755 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:24.439710 2565 scope.go:117] "RemoveContainer" containerID="dcb40b6e610f19d5a48296dafb1829ca6cd19dfeb82c0ad722b6ec8f6a075a47" Jul 28 01:48:24.448199 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:24.448183 2565 scope.go:117] "RemoveContainer" containerID="dcb40b6e610f19d5a48296dafb1829ca6cd19dfeb82c0ad722b6ec8f6a075a47" Jul 28 01:48:24.448473 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:48:24.448456 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcb40b6e610f19d5a48296dafb1829ca6cd19dfeb82c0ad722b6ec8f6a075a47\": container with ID starting with dcb40b6e610f19d5a48296dafb1829ca6cd19dfeb82c0ad722b6ec8f6a075a47 not found: ID does not exist" containerID="dcb40b6e610f19d5a48296dafb1829ca6cd19dfeb82c0ad722b6ec8f6a075a47" Jul 28 01:48:24.448543 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:24.448486 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcb40b6e610f19d5a48296dafb1829ca6cd19dfeb82c0ad722b6ec8f6a075a47"} err="failed to get container status \"dcb40b6e610f19d5a48296dafb1829ca6cd19dfeb82c0ad722b6ec8f6a075a47\": rpc error: code = NotFound desc = could not find container \"dcb40b6e610f19d5a48296dafb1829ca6cd19dfeb82c0ad722b6ec8f6a075a47\": container with ID starting with dcb40b6e610f19d5a48296dafb1829ca6cd19dfeb82c0ad722b6ec8f6a075a47 not found: ID does not exist" Jul 28 01:48:24.468840 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:24.468813 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6037fca-f880-4656-86be-a3bd0c003dad-openshift-service-ca\") pod \"c6037fca-f880-4656-86be-a3bd0c003dad\" (UID: \"c6037fca-f880-4656-86be-a3bd0c003dad\") " Jul 28 01:48:24.468976 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:24.468887 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7tv5\" (UniqueName: \"kubernetes.io/projected/c6037fca-f880-4656-86be-a3bd0c003dad-kube-api-access-r7tv5\") pod \"c6037fca-f880-4656-86be-a3bd0c003dad\" (UID: \"c6037fca-f880-4656-86be-a3bd0c003dad\") " Jul 28 01:48:24.468976 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:24.468917 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6037fca-f880-4656-86be-a3bd0c003dad-maas-api-tls\") pod \"c6037fca-f880-4656-86be-a3bd0c003dad\" (UID: \"c6037fca-f880-4656-86be-a3bd0c003dad\") " Jul 28 01:48:24.469272 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:24.469233 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6037fca-f880-4656-86be-a3bd0c003dad-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c6037fca-f880-4656-86be-a3bd0c003dad" (UID: "c6037fca-f880-4656-86be-a3bd0c003dad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:48:24.471196 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:24.471170 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6037fca-f880-4656-86be-a3bd0c003dad-kube-api-access-r7tv5" (OuterVolumeSpecName: "kube-api-access-r7tv5") pod "c6037fca-f880-4656-86be-a3bd0c003dad" (UID: "c6037fca-f880-4656-86be-a3bd0c003dad"). InnerVolumeSpecName "kube-api-access-r7tv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:48:24.471297 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:24.471202 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6037fca-f880-4656-86be-a3bd0c003dad-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c6037fca-f880-4656-86be-a3bd0c003dad" (UID: "c6037fca-f880-4656-86be-a3bd0c003dad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:48:24.570573 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:24.570536 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6037fca-f880-4656-86be-a3bd0c003dad-maas-api-tls\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:48:24.570573 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:24.570569 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6037fca-f880-4656-86be-a3bd0c003dad-openshift-service-ca\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:48:24.570573 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:24.570579 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7tv5\" (UniqueName: \"kubernetes.io/projected/c6037fca-f880-4656-86be-a3bd0c003dad-kube-api-access-r7tv5\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:48:24.761776 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:24.761741 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn"] Jul 28 01:48:24.765562 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:24.765535 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn"] Jul 28 01:48:25.101061 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:25.100964 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6037fca-f880-4656-86be-a3bd0c003dad" path="/var/lib/kubelet/pods/c6037fca-f880-4656-86be-a3bd0c003dad/volumes" Jul 28 01:48:27.328796 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:27.328753 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-43a6ae00-7ff5df99c9-w56tn" podUID="c6037fca-f880-4656-86be-a3bd0c003dad" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.44:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 28 01:48:28.768896 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:28.768863 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-43a6ae00-xrrnm"] Jul 28 01:48:28.770704 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:28.770682 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-43a6ae00-xrrnm"] Jul 28 01:48:29.100527 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:29.100441 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99253286-389e-46ca-b331-8ce6bb5ba28a" path="/var/lib/kubelet/pods/99253286-389e-46ca-b331-8ce6bb5ba28a/volumes" Jul 28 01:48:57.308446 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:57.308410 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt"] Jul 28 01:48:57.308911 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:57.308723 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99253286-389e-46ca-b331-8ce6bb5ba28a" containerName="revoke-keys" Jul 28 01:48:57.308911 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:57.308735 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="99253286-389e-46ca-b331-8ce6bb5ba28a" containerName="revoke-keys" Jul 28 01:48:57.308911 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:57.308753 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6037fca-f880-4656-86be-a3bd0c003dad" containerName="maas-api" Jul 28 01:48:57.308911 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:57.308758 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6037fca-f880-4656-86be-a3bd0c003dad" containerName="maas-api" Jul 28 01:48:57.308911 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:57.308808 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6037fca-f880-4656-86be-a3bd0c003dad" containerName="maas-api" Jul 28 01:48:57.308911 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:57.308817 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="99253286-389e-46ca-b331-8ce6bb5ba28a" containerName="revoke-keys" Jul 28 01:48:57.311841 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:57.311821 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt" Jul 28 01:48:57.314622 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:57.314599 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2a202a1f-serving-cert\"" Jul 28 01:48:57.314622 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:57.314612 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-62vvq\"" Jul 28 01:48:57.315698 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:57.315655 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 28 01:48:57.315809 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:57.315717 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 01:48:57.325745 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:57.325720 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt"] Jul 28 01:48:57.349998 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:57.349961 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8247a6d6-77bb-4120-9031-5773f9f43df1-maas-api-tls\") pod \"maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt\" (UID: \"8247a6d6-77bb-4120-9031-5773f9f43df1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt" Jul 28 01:48:57.350198 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:57.350015 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqm6h\" (UniqueName: \"kubernetes.io/projected/8247a6d6-77bb-4120-9031-5773f9f43df1-kube-api-access-gqm6h\") pod \"maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt\" (UID: \"8247a6d6-77bb-4120-9031-5773f9f43df1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt" Jul 28 01:48:57.350198 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:57.350091 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8247a6d6-77bb-4120-9031-5773f9f43df1-openshift-service-ca\") pod \"maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt\" (UID: \"8247a6d6-77bb-4120-9031-5773f9f43df1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt" Jul 28 01:48:57.450540 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:57.450494 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqm6h\" (UniqueName: \"kubernetes.io/projected/8247a6d6-77bb-4120-9031-5773f9f43df1-kube-api-access-gqm6h\") pod \"maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt\" (UID: \"8247a6d6-77bb-4120-9031-5773f9f43df1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt" Jul 28 01:48:57.450731 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:57.450545 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8247a6d6-77bb-4120-9031-5773f9f43df1-openshift-service-ca\") pod \"maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt\" (UID: \"8247a6d6-77bb-4120-9031-5773f9f43df1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt" Jul 28 01:48:57.450731 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:57.450617 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8247a6d6-77bb-4120-9031-5773f9f43df1-maas-api-tls\") pod \"maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt\" (UID: \"8247a6d6-77bb-4120-9031-5773f9f43df1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt" Jul 28 01:48:57.450826 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:48:57.450733 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-2a202a1f-serving-cert: secret "maas-api-e2e-mt-2a202a1f-serving-cert" not found Jul 28 01:48:57.450826 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:48:57.450806 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8247a6d6-77bb-4120-9031-5773f9f43df1-maas-api-tls podName:8247a6d6-77bb-4120-9031-5773f9f43df1 nodeName:}" failed. No retries permitted until 2026-07-28 01:48:57.950786472 +0000 UTC m=+1995.385184764 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8247a6d6-77bb-4120-9031-5773f9f43df1-maas-api-tls") pod "maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt" (UID: "8247a6d6-77bb-4120-9031-5773f9f43df1") : secret "maas-api-e2e-mt-2a202a1f-serving-cert" not found Jul 28 01:48:57.451294 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:57.451273 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8247a6d6-77bb-4120-9031-5773f9f43df1-openshift-service-ca\") pod \"maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt\" (UID: \"8247a6d6-77bb-4120-9031-5773f9f43df1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt" Jul 28 01:48:57.459139 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:57.459110 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqm6h\" (UniqueName: \"kubernetes.io/projected/8247a6d6-77bb-4120-9031-5773f9f43df1-kube-api-access-gqm6h\") pod \"maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt\" (UID: \"8247a6d6-77bb-4120-9031-5773f9f43df1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt" Jul 28 01:48:57.954602 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:57.954563 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8247a6d6-77bb-4120-9031-5773f9f43df1-maas-api-tls\") pod \"maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt\" (UID: \"8247a6d6-77bb-4120-9031-5773f9f43df1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt" Jul 28 01:48:57.957140 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:57.957110 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8247a6d6-77bb-4120-9031-5773f9f43df1-maas-api-tls\") pod \"maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt\" (UID: \"8247a6d6-77bb-4120-9031-5773f9f43df1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt" Jul 28 01:48:58.222444 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:58.222343 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt" Jul 28 01:48:58.354228 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:58.354204 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt"] Jul 28 01:48:58.356299 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:48:58.356259 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8247a6d6_77bb_4120_9031_5773f9f43df1.slice/crio-8efd4b23a796f488e137de919c10335b641a906432cc7cfa7676cc80683183fc WatchSource:0}: Error finding container 8efd4b23a796f488e137de919c10335b641a906432cc7cfa7676cc80683183fc: Status 404 returned error can't find the container with id 8efd4b23a796f488e137de919c10335b641a906432cc7cfa7676cc80683183fc Jul 28 01:48:58.357735 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:58.357718 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 01:48:58.615933 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:58.615840 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt" event={"ID":"8247a6d6-77bb-4120-9031-5773f9f43df1","Type":"ContainerStarted","Data":"8efd4b23a796f488e137de919c10335b641a906432cc7cfa7676cc80683183fc"} Jul 28 01:48:59.621790 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:59.621756 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt" event={"ID":"8247a6d6-77bb-4120-9031-5773f9f43df1","Type":"ContainerStarted","Data":"6d75788d135280c39a389e8e1e40d702b45507aca555b50a9e8e0d23e0a3378e"} Jul 28 01:48:59.622228 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:59.621847 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt" Jul 28 01:48:59.637957 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:48:59.637908 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt" podStartSLOduration=2.272256872 podStartE2EDuration="2.637892599s" totalImagesPullingTime="365.635727ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:48:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:48:59.637234836 +0000 UTC m=+1997.071633153" watchObservedRunningTime="2026-07-28 01:48:59.637892599 +0000 UTC m=+1997.072290913" Jul 28 01:49:05.633168 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:49:05.633135 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt" Jul 28 01:49:15.809873 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:49:15.809839 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c"] Jul 28 01:49:15.813443 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:49:15.813424 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c" Jul 28 01:49:15.817288 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:49:15.817266 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2a202a1f\"/\"default-dockercfg-jv8mv\"" Jul 28 01:49:15.817407 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:49:15.817341 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2a202a1f\"/\"openshift-service-ca.crt\"" Jul 28 01:49:15.817407 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:49:15.817360 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2a202a1f\"/\"e2e-lifecycle-model-2a202a1f-kserve-self-signed-certs\"" Jul 28 01:49:15.817407 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:49:15.817285 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2a202a1f\"/\"kube-root-ca.crt\"" Jul 28 01:49:15.821556 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:49:15.821533 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c"] Jul 28 01:49:15.907218 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:49:15.907177 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09f4daff-0699-481a-ac4b-76e4b4ee9a8e-dshm\") pod \"e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c\" (UID: \"09f4daff-0699-481a-ac4b-76e4b4ee9a8e\") " pod="ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c" Jul 28 01:49:15.907218 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:49:15.907221 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2gqx\" (UniqueName: \"kubernetes.io/projected/09f4daff-0699-481a-ac4b-76e4b4ee9a8e-kube-api-access-q2gqx\") pod \"e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c\" (UID: \"09f4daff-0699-481a-ac4b-76e4b4ee9a8e\") " pod="ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c" Jul 28 01:49:15.907441 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:49:15.907249 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09f4daff-0699-481a-ac4b-76e4b4ee9a8e-tls-certs\") pod \"e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c\" (UID: \"09f4daff-0699-481a-ac4b-76e4b4ee9a8e\") " pod="ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c" Jul 28 01:49:15.907441 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:49:15.907314 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09f4daff-0699-481a-ac4b-76e4b4ee9a8e-home\") pod \"e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c\" (UID: \"09f4daff-0699-481a-ac4b-76e4b4ee9a8e\") " pod="ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c" Jul 28 01:49:15.907441 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:49:15.907353 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09f4daff-0699-481a-ac4b-76e4b4ee9a8e-tmp-dir\") pod \"e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c\" (UID: \"09f4daff-0699-481a-ac4b-76e4b4ee9a8e\") " pod="ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c" Jul 28 01:49:15.907441 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:49:15.907390 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09f4daff-0699-481a-ac4b-76e4b4ee9a8e-model-cache\") pod \"e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c\" (UID: \"09f4daff-0699-481a-ac4b-76e4b4ee9a8e\") " pod="ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c" Jul 28 01:49:16.008107 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:49:16.008062 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09f4daff-0699-481a-ac4b-76e4b4ee9a8e-dshm\") pod \"e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c\" (UID: \"09f4daff-0699-481a-ac4b-76e4b4ee9a8e\") " pod="ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c" Jul 28 01:49:16.008107 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:49:16.008104 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2gqx\" (UniqueName: \"kubernetes.io/projected/09f4daff-0699-481a-ac4b-76e4b4ee9a8e-kube-api-access-q2gqx\") pod \"e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c\" (UID: \"09f4daff-0699-481a-ac4b-76e4b4ee9a8e\") " pod="ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c" Jul 28 01:49:16.008344 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:49:16.008123 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09f4daff-0699-481a-ac4b-76e4b4ee9a8e-tls-certs\") pod \"e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c\" (UID: \"09f4daff-0699-481a-ac4b-76e4b4ee9a8e\") " pod="ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c" Jul 28 01:49:16.008344 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:49:16.008307 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09f4daff-0699-481a-ac4b-76e4b4ee9a8e-home\") pod \"e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c\" (UID: \"09f4daff-0699-481a-ac4b-76e4b4ee9a8e\") " pod="ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c" Jul 28 01:49:16.008454 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:49:16.008352 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09f4daff-0699-481a-ac4b-76e4b4ee9a8e-tmp-dir\") pod \"e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c\" (UID: \"09f4daff-0699-481a-ac4b-76e4b4ee9a8e\") " pod="ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c" Jul 28 01:49:16.008454 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:49:16.008411 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09f4daff-0699-481a-ac4b-76e4b4ee9a8e-model-cache\") pod \"e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c\" (UID: \"09f4daff-0699-481a-ac4b-76e4b4ee9a8e\") " pod="ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c" Jul 28 01:49:16.008692 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:49:16.008663 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09f4daff-0699-481a-ac4b-76e4b4ee9a8e-home\") pod \"e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c\" (UID: \"09f4daff-0699-481a-ac4b-76e4b4ee9a8e\") " pod="ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c" Jul 28 01:49:16.008818 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:49:16.008723 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09f4daff-0699-481a-ac4b-76e4b4ee9a8e-tmp-dir\") pod \"e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c\" (UID: \"09f4daff-0699-481a-ac4b-76e4b4ee9a8e\") " pod="ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c" Jul 28 01:49:16.008818 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:49:16.008797 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09f4daff-0699-481a-ac4b-76e4b4ee9a8e-model-cache\") pod \"e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c\" (UID: \"09f4daff-0699-481a-ac4b-76e4b4ee9a8e\") " pod="ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c" Jul 28 01:49:16.010389 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:49:16.010374 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09f4daff-0699-481a-ac4b-76e4b4ee9a8e-dshm\") pod \"e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c\" (UID: \"09f4daff-0699-481a-ac4b-76e4b4ee9a8e\") " pod="ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c" Jul 28 01:49:16.010669 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:49:16.010654 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09f4daff-0699-481a-ac4b-76e4b4ee9a8e-tls-certs\") pod \"e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c\" (UID: \"09f4daff-0699-481a-ac4b-76e4b4ee9a8e\") " pod="ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c" Jul 28 01:49:16.015741 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:49:16.015723 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2gqx\" (UniqueName: \"kubernetes.io/projected/09f4daff-0699-481a-ac4b-76e4b4ee9a8e-kube-api-access-q2gqx\") pod \"e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c\" (UID: \"09f4daff-0699-481a-ac4b-76e4b4ee9a8e\") " pod="ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c" Jul 28 01:49:16.123273 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:49:16.123174 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c" Jul 28 01:49:16.258156 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:49:16.258128 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c"] Jul 28 01:49:16.259919 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:49:16.259895 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09f4daff_0699_481a_ac4b_76e4b4ee9a8e.slice/crio-7dc142d21a9fbbcdd17285e995b0058ba56afbd449731333cf7e8b82f1eae7a5 WatchSource:0}: Error finding container 7dc142d21a9fbbcdd17285e995b0058ba56afbd449731333cf7e8b82f1eae7a5: Status 404 returned error can't find the container with id 7dc142d21a9fbbcdd17285e995b0058ba56afbd449731333cf7e8b82f1eae7a5 Jul 28 01:49:16.708495 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:49:16.708455 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c" event={"ID":"09f4daff-0699-481a-ac4b-76e4b4ee9a8e","Type":"ContainerStarted","Data":"795f8e1a39540cb6fabc0d3f8a0be382edb20f522b4ae4c757a309e7a54296a8"} Jul 28 01:49:16.708689 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:49:16.708507 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c" event={"ID":"09f4daff-0699-481a-ac4b-76e4b4ee9a8e","Type":"ContainerStarted","Data":"7dc142d21a9fbbcdd17285e995b0058ba56afbd449731333cf7e8b82f1eae7a5"} Jul 28 01:49:16.729070 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:49:16.728983 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c" podStartSLOduration=1.728961073 podStartE2EDuration="1.728961073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:49:16.725342101 +0000 UTC m=+2014.159740418" watchObservedRunningTime="2026-07-28 01:49:16.728961073 +0000 UTC m=+2014.163359387" Jul 28 01:49:26.123328 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:49:26.123289 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c" Jul 28 01:49:26.123943 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:49:26.123346 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c" Jul 28 01:49:26.135617 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:49:26.135591 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c" Jul 28 01:49:26.769283 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:49:26.769248 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c" Jul 28 01:50:07.118318 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:07.118228 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt"] Jul 28 01:50:07.118973 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:07.118472 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt" podUID="8247a6d6-77bb-4120-9031-5773f9f43df1" containerName="maas-api" containerID="cri-o://6d75788d135280c39a389e8e1e40d702b45507aca555b50a9e8e0d23e0a3378e" gracePeriod=30 Jul 28 01:50:07.379284 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:07.379223 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt" Jul 28 01:50:07.487374 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:07.487342 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqm6h\" (UniqueName: \"kubernetes.io/projected/8247a6d6-77bb-4120-9031-5773f9f43df1-kube-api-access-gqm6h\") pod \"8247a6d6-77bb-4120-9031-5773f9f43df1\" (UID: \"8247a6d6-77bb-4120-9031-5773f9f43df1\") " Jul 28 01:50:07.487374 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:07.487378 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8247a6d6-77bb-4120-9031-5773f9f43df1-openshift-service-ca\") pod \"8247a6d6-77bb-4120-9031-5773f9f43df1\" (UID: \"8247a6d6-77bb-4120-9031-5773f9f43df1\") " Jul 28 01:50:07.487628 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:07.487414 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8247a6d6-77bb-4120-9031-5773f9f43df1-maas-api-tls\") pod \"8247a6d6-77bb-4120-9031-5773f9f43df1\" (UID: \"8247a6d6-77bb-4120-9031-5773f9f43df1\") " Jul 28 01:50:07.487845 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:07.487822 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8247a6d6-77bb-4120-9031-5773f9f43df1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8247a6d6-77bb-4120-9031-5773f9f43df1" (UID: "8247a6d6-77bb-4120-9031-5773f9f43df1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:50:07.489546 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:07.489514 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8247a6d6-77bb-4120-9031-5773f9f43df1-kube-api-access-gqm6h" (OuterVolumeSpecName: "kube-api-access-gqm6h") pod "8247a6d6-77bb-4120-9031-5773f9f43df1" (UID: "8247a6d6-77bb-4120-9031-5773f9f43df1"). InnerVolumeSpecName "kube-api-access-gqm6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:50:07.489680 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:07.489563 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8247a6d6-77bb-4120-9031-5773f9f43df1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8247a6d6-77bb-4120-9031-5773f9f43df1" (UID: "8247a6d6-77bb-4120-9031-5773f9f43df1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:50:07.588213 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:07.588173 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqm6h\" (UniqueName: \"kubernetes.io/projected/8247a6d6-77bb-4120-9031-5773f9f43df1-kube-api-access-gqm6h\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:50:07.588213 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:07.588206 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8247a6d6-77bb-4120-9031-5773f9f43df1-openshift-service-ca\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:50:07.588213 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:07.588220 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8247a6d6-77bb-4120-9031-5773f9f43df1-maas-api-tls\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:50:07.968108 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:07.968075 2565 generic.go:356] "Generic (PLEG): container finished" podID="8247a6d6-77bb-4120-9031-5773f9f43df1" containerID="6d75788d135280c39a389e8e1e40d702b45507aca555b50a9e8e0d23e0a3378e" exitCode=0 Jul 28 01:50:07.968303 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:07.968163 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt" Jul 28 01:50:07.968303 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:07.968170 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt" event={"ID":"8247a6d6-77bb-4120-9031-5773f9f43df1","Type":"ContainerDied","Data":"6d75788d135280c39a389e8e1e40d702b45507aca555b50a9e8e0d23e0a3378e"} Jul 28 01:50:07.968303 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:07.968226 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt" event={"ID":"8247a6d6-77bb-4120-9031-5773f9f43df1","Type":"ContainerDied","Data":"8efd4b23a796f488e137de919c10335b641a906432cc7cfa7676cc80683183fc"} Jul 28 01:50:07.968303 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:07.968267 2565 scope.go:117] "RemoveContainer" containerID="6d75788d135280c39a389e8e1e40d702b45507aca555b50a9e8e0d23e0a3378e" Jul 28 01:50:07.977537 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:07.977510 2565 scope.go:117] "RemoveContainer" containerID="6d75788d135280c39a389e8e1e40d702b45507aca555b50a9e8e0d23e0a3378e" Jul 28 01:50:07.977804 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:50:07.977785 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d75788d135280c39a389e8e1e40d702b45507aca555b50a9e8e0d23e0a3378e\": container with ID starting with 6d75788d135280c39a389e8e1e40d702b45507aca555b50a9e8e0d23e0a3378e not found: ID does not exist" containerID="6d75788d135280c39a389e8e1e40d702b45507aca555b50a9e8e0d23e0a3378e" Jul 28 01:50:07.977855 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:07.977813 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d75788d135280c39a389e8e1e40d702b45507aca555b50a9e8e0d23e0a3378e"} err="failed to get container status \"6d75788d135280c39a389e8e1e40d702b45507aca555b50a9e8e0d23e0a3378e\": rpc error: code = NotFound desc = could not find container \"6d75788d135280c39a389e8e1e40d702b45507aca555b50a9e8e0d23e0a3378e\": container with ID starting with 6d75788d135280c39a389e8e1e40d702b45507aca555b50a9e8e0d23e0a3378e not found: ID does not exist" Jul 28 01:50:07.990168 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:07.990138 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt"] Jul 28 01:50:07.992648 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:07.992622 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a202a1f-77487bd658-7k2mt"] Jul 28 01:50:09.100671 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:09.100638 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8247a6d6-77bb-4120-9031-5773f9f43df1" path="/var/lib/kubelet/pods/8247a6d6-77bb-4120-9031-5773f9f43df1/volumes" Jul 28 01:50:20.015399 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:20.015361 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c"] Jul 28 01:50:35.035277 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:35.035212 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c" podUID="09f4daff-0699-481a-ac4b-76e4b4ee9a8e" containerName="main" containerID="cri-o://795f8e1a39540cb6fabc0d3f8a0be382edb20f522b4ae4c757a309e7a54296a8" gracePeriod=45 Jul 28 01:50:35.278374 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:35.278349 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c" Jul 28 01:50:35.434312 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:35.434215 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09f4daff-0699-481a-ac4b-76e4b4ee9a8e-tls-certs\") pod \"09f4daff-0699-481a-ac4b-76e4b4ee9a8e\" (UID: \"09f4daff-0699-481a-ac4b-76e4b4ee9a8e\") " Jul 28 01:50:35.434312 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:35.434277 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09f4daff-0699-481a-ac4b-76e4b4ee9a8e-dshm\") pod \"09f4daff-0699-481a-ac4b-76e4b4ee9a8e\" (UID: \"09f4daff-0699-481a-ac4b-76e4b4ee9a8e\") " Jul 28 01:50:35.434312 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:35.434305 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09f4daff-0699-481a-ac4b-76e4b4ee9a8e-home\") pod \"09f4daff-0699-481a-ac4b-76e4b4ee9a8e\" (UID: \"09f4daff-0699-481a-ac4b-76e4b4ee9a8e\") " Jul 28 01:50:35.434597 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:35.434325 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2gqx\" (UniqueName: \"kubernetes.io/projected/09f4daff-0699-481a-ac4b-76e4b4ee9a8e-kube-api-access-q2gqx\") pod \"09f4daff-0699-481a-ac4b-76e4b4ee9a8e\" (UID: \"09f4daff-0699-481a-ac4b-76e4b4ee9a8e\") " Jul 28 01:50:35.434597 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:35.434348 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09f4daff-0699-481a-ac4b-76e4b4ee9a8e-model-cache\") pod \"09f4daff-0699-481a-ac4b-76e4b4ee9a8e\" (UID: \"09f4daff-0699-481a-ac4b-76e4b4ee9a8e\") " Jul 28 01:50:35.434597 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:35.434387 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09f4daff-0699-481a-ac4b-76e4b4ee9a8e-tmp-dir\") pod \"09f4daff-0699-481a-ac4b-76e4b4ee9a8e\" (UID: \"09f4daff-0699-481a-ac4b-76e4b4ee9a8e\") " Jul 28 01:50:35.434597 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:35.434571 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f4daff-0699-481a-ac4b-76e4b4ee9a8e-home" (OuterVolumeSpecName: "home") pod "09f4daff-0699-481a-ac4b-76e4b4ee9a8e" (UID: "09f4daff-0699-481a-ac4b-76e4b4ee9a8e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:50:35.434796 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:35.434711 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f4daff-0699-481a-ac4b-76e4b4ee9a8e-model-cache" (OuterVolumeSpecName: "model-cache") pod "09f4daff-0699-481a-ac4b-76e4b4ee9a8e" (UID: "09f4daff-0699-481a-ac4b-76e4b4ee9a8e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:50:35.434796 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:35.434726 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f4daff-0699-481a-ac4b-76e4b4ee9a8e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "09f4daff-0699-481a-ac4b-76e4b4ee9a8e" (UID: "09f4daff-0699-481a-ac4b-76e4b4ee9a8e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:50:35.436522 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:35.436496 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f4daff-0699-481a-ac4b-76e4b4ee9a8e-dshm" (OuterVolumeSpecName: "dshm") pod "09f4daff-0699-481a-ac4b-76e4b4ee9a8e" (UID: "09f4daff-0699-481a-ac4b-76e4b4ee9a8e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:50:35.436522 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:35.436501 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09f4daff-0699-481a-ac4b-76e4b4ee9a8e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "09f4daff-0699-481a-ac4b-76e4b4ee9a8e" (UID: "09f4daff-0699-481a-ac4b-76e4b4ee9a8e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:50:35.436666 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:35.436526 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f4daff-0699-481a-ac4b-76e4b4ee9a8e-kube-api-access-q2gqx" (OuterVolumeSpecName: "kube-api-access-q2gqx") pod "09f4daff-0699-481a-ac4b-76e4b4ee9a8e" (UID: "09f4daff-0699-481a-ac4b-76e4b4ee9a8e"). InnerVolumeSpecName "kube-api-access-q2gqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:50:35.534963 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:35.534934 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09f4daff-0699-481a-ac4b-76e4b4ee9a8e-tmp-dir\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:50:35.534963 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:35.534961 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09f4daff-0699-481a-ac4b-76e4b4ee9a8e-tls-certs\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:50:35.534963 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:35.534969 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09f4daff-0699-481a-ac4b-76e4b4ee9a8e-dshm\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:50:35.535229 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:35.534977 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09f4daff-0699-481a-ac4b-76e4b4ee9a8e-home\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:50:35.535229 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:35.534985 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2gqx\" (UniqueName: \"kubernetes.io/projected/09f4daff-0699-481a-ac4b-76e4b4ee9a8e-kube-api-access-q2gqx\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:50:35.535229 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:35.534993 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09f4daff-0699-481a-ac4b-76e4b4ee9a8e-model-cache\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:50:36.110886 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:36.110849 2565 generic.go:356] "Generic (PLEG): container finished" podID="09f4daff-0699-481a-ac4b-76e4b4ee9a8e" containerID="795f8e1a39540cb6fabc0d3f8a0be382edb20f522b4ae4c757a309e7a54296a8" exitCode=0 Jul 28 01:50:36.111408 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:36.110894 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c" Jul 28 01:50:36.111408 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:36.110930 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c" event={"ID":"09f4daff-0699-481a-ac4b-76e4b4ee9a8e","Type":"ContainerDied","Data":"795f8e1a39540cb6fabc0d3f8a0be382edb20f522b4ae4c757a309e7a54296a8"} Jul 28 01:50:36.111408 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:36.110971 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c" event={"ID":"09f4daff-0699-481a-ac4b-76e4b4ee9a8e","Type":"ContainerDied","Data":"7dc142d21a9fbbcdd17285e995b0058ba56afbd449731333cf7e8b82f1eae7a5"} Jul 28 01:50:36.111408 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:36.110984 2565 scope.go:117] "RemoveContainer" containerID="795f8e1a39540cb6fabc0d3f8a0be382edb20f522b4ae4c757a309e7a54296a8" Jul 28 01:50:36.120822 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:36.120807 2565 scope.go:117] "RemoveContainer" containerID="795f8e1a39540cb6fabc0d3f8a0be382edb20f522b4ae4c757a309e7a54296a8" Jul 28 01:50:36.121083 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:50:36.121060 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"795f8e1a39540cb6fabc0d3f8a0be382edb20f522b4ae4c757a309e7a54296a8\": container with ID starting with 795f8e1a39540cb6fabc0d3f8a0be382edb20f522b4ae4c757a309e7a54296a8 not found: ID does not exist" containerID="795f8e1a39540cb6fabc0d3f8a0be382edb20f522b4ae4c757a309e7a54296a8" Jul 28 01:50:36.121165 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:36.121091 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"795f8e1a39540cb6fabc0d3f8a0be382edb20f522b4ae4c757a309e7a54296a8"} err="failed to get container status \"795f8e1a39540cb6fabc0d3f8a0be382edb20f522b4ae4c757a309e7a54296a8\": rpc error: code = NotFound desc = could not find container \"795f8e1a39540cb6fabc0d3f8a0be382edb20f522b4ae4c757a309e7a54296a8\": container with ID starting with 795f8e1a39540cb6fabc0d3f8a0be382edb20f522b4ae4c757a309e7a54296a8 not found: ID does not exist" Jul 28 01:50:36.133436 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:36.133412 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c"] Jul 28 01:50:36.140166 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:36.140144 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-2a202a1f/e2e-lifecycle-model-2a202a1f-kserve-6dbd845db7-gs84c"] Jul 28 01:50:37.101178 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:50:37.101143 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09f4daff-0699-481a-ac4b-76e4b4ee9a8e" path="/var/lib/kubelet/pods/09f4daff-0699-481a-ac4b-76e4b4ee9a8e/volumes" Jul 28 01:51:43.211000 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:51:43.210970 2565 scope.go:117] "RemoveContainer" containerID="42603dd8a049e0ff7d278dbb39f21d570aebaf236c648b593498991a31d78ba8" Jul 28 01:51:43.219150 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:51:43.219129 2565 scope.go:117] "RemoveContainer" containerID="3c3fdf2bcde14a00a59b5078adb95ed8f87c4ef056da9a09fee0448488bf12f4" Jul 28 01:52:14.505445 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:14.505409 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4"] Jul 28 01:52:14.505912 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:14.505730 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09f4daff-0699-481a-ac4b-76e4b4ee9a8e" containerName="main" Jul 28 01:52:14.505912 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:14.505743 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="09f4daff-0699-481a-ac4b-76e4b4ee9a8e" containerName="main" Jul 28 01:52:14.505912 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:14.505759 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8247a6d6-77bb-4120-9031-5773f9f43df1" containerName="maas-api" Jul 28 01:52:14.505912 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:14.505765 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8247a6d6-77bb-4120-9031-5773f9f43df1" containerName="maas-api" Jul 28 01:52:14.505912 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:14.505819 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="09f4daff-0699-481a-ac4b-76e4b4ee9a8e" containerName="main" Jul 28 01:52:14.505912 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:14.505829 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8247a6d6-77bb-4120-9031-5773f9f43df1" containerName="maas-api" Jul 28 01:52:14.508883 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:14.508863 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4" Jul 28 01:52:14.511452 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:14.511429 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 28 01:52:14.511452 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:14.511437 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-896d36-serving-cert\"" Jul 28 01:52:14.512445 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:14.512424 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 01:52:14.512550 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:14.512424 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-62vvq\"" Jul 28 01:52:14.518182 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:14.518156 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4"] Jul 28 01:52:14.571834 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:14.571800 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/249144ab-463b-4e3a-80f1-4c41171f912f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4\" (UID: \"249144ab-463b-4e3a-80f1-4c41171f912f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4" Jul 28 01:52:14.571980 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:14.571846 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/249144ab-463b-4e3a-80f1-4c41171f912f-maas-api-tls\") pod \"maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4\" (UID: \"249144ab-463b-4e3a-80f1-4c41171f912f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4" Jul 28 01:52:14.571980 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:14.571873 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bft5\" (UniqueName: \"kubernetes.io/projected/249144ab-463b-4e3a-80f1-4c41171f912f-kube-api-access-2bft5\") pod \"maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4\" (UID: \"249144ab-463b-4e3a-80f1-4c41171f912f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4" Jul 28 01:52:14.672319 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:14.672289 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/249144ab-463b-4e3a-80f1-4c41171f912f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4\" (UID: \"249144ab-463b-4e3a-80f1-4c41171f912f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4" Jul 28 01:52:14.672319 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:14.672324 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/249144ab-463b-4e3a-80f1-4c41171f912f-maas-api-tls\") pod \"maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4\" (UID: \"249144ab-463b-4e3a-80f1-4c41171f912f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4" Jul 28 01:52:14.672528 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:14.672351 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bft5\" (UniqueName: \"kubernetes.io/projected/249144ab-463b-4e3a-80f1-4c41171f912f-kube-api-access-2bft5\") pod \"maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4\" (UID: \"249144ab-463b-4e3a-80f1-4c41171f912f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4" Jul 28 01:52:14.672528 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:52:14.672455 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-896d36-serving-cert: secret "maas-api-e2e-inf-a-896d36-serving-cert" not found Jul 28 01:52:14.672602 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:52:14.672529 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/249144ab-463b-4e3a-80f1-4c41171f912f-maas-api-tls podName:249144ab-463b-4e3a-80f1-4c41171f912f nodeName:}" failed. No retries permitted until 2026-07-28 01:52:15.172508888 +0000 UTC m=+2192.606907199 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/249144ab-463b-4e3a-80f1-4c41171f912f-maas-api-tls") pod "maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4" (UID: "249144ab-463b-4e3a-80f1-4c41171f912f") : secret "maas-api-e2e-inf-a-896d36-serving-cert" not found Jul 28 01:52:14.673028 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:14.673011 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/249144ab-463b-4e3a-80f1-4c41171f912f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4\" (UID: \"249144ab-463b-4e3a-80f1-4c41171f912f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4" Jul 28 01:52:14.683249 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:14.683226 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bft5\" (UniqueName: \"kubernetes.io/projected/249144ab-463b-4e3a-80f1-4c41171f912f-kube-api-access-2bft5\") pod \"maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4\" (UID: \"249144ab-463b-4e3a-80f1-4c41171f912f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4" Jul 28 01:52:15.177640 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:15.177605 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/249144ab-463b-4e3a-80f1-4c41171f912f-maas-api-tls\") pod \"maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4\" (UID: \"249144ab-463b-4e3a-80f1-4c41171f912f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4" Jul 28 01:52:15.179971 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:15.179942 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/249144ab-463b-4e3a-80f1-4c41171f912f-maas-api-tls\") pod \"maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4\" (UID: \"249144ab-463b-4e3a-80f1-4c41171f912f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4" Jul 28 01:52:15.422241 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:15.422211 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4" Jul 28 01:52:15.555739 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:15.555648 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4"] Jul 28 01:52:15.623013 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:15.622980 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4" event={"ID":"249144ab-463b-4e3a-80f1-4c41171f912f","Type":"ContainerStarted","Data":"a7c443af80d74c8d80c42da599e3ccdd1463a0928c3cb57181dc094869a5d469"} Jul 28 01:52:16.629528 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:16.629486 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4" event={"ID":"249144ab-463b-4e3a-80f1-4c41171f912f","Type":"ContainerStarted","Data":"66b24ec0a2c3a370c7f2a3e94d08479c02ce6cd2fab35968f021948d9d819540"} Jul 28 01:52:16.629945 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:16.629628 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4" Jul 28 01:52:16.645811 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:16.645764 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4" podStartSLOduration=2.245769603 podStartE2EDuration="2.645749463s" totalImagesPullingTime="399.97986ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:52:16.645359941 +0000 UTC m=+2194.079758256" watchObservedRunningTime="2026-07-28 01:52:16.645749463 +0000 UTC m=+2194.080147777" Jul 28 01:52:19.463651 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:19.463613 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8"] Jul 28 01:52:19.467011 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:19.466984 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:19.469562 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:19.469540 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-afe8f9-openshift-default-dockercfg-fpslx\"" Jul 28 01:52:19.480426 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:19.480402 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8"] Jul 28 01:52:19.514639 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:19.514601 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1931c057-ff82-463e-afa9-5cc33d43f082-istio-data\") pod \"e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8\" (UID: \"1931c057-ff82-463e-afa9-5cc33d43f082\") " pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:19.514800 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:19.514669 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1931c057-ff82-463e-afa9-5cc33d43f082-credential-socket\") pod \"e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8\" (UID: \"1931c057-ff82-463e-afa9-5cc33d43f082\") " pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:19.514800 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:19.514726 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1931c057-ff82-463e-afa9-5cc33d43f082-workload-socket\") pod \"e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8\" (UID: \"1931c057-ff82-463e-afa9-5cc33d43f082\") " pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:19.514800 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:19.514783 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jml2d\" (UniqueName: \"kubernetes.io/projected/1931c057-ff82-463e-afa9-5cc33d43f082-kube-api-access-jml2d\") pod \"e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8\" (UID: \"1931c057-ff82-463e-afa9-5cc33d43f082\") " pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:19.514907 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:19.514815 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1931c057-ff82-463e-afa9-5cc33d43f082-istio-token\") pod \"e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8\" (UID: \"1931c057-ff82-463e-afa9-5cc33d43f082\") " pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:19.514907 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:19.514842 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1931c057-ff82-463e-afa9-5cc33d43f082-istiod-ca-cert\") pod \"e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8\" (UID: \"1931c057-ff82-463e-afa9-5cc33d43f082\") " pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:19.514907 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:19.514869 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1931c057-ff82-463e-afa9-5cc33d43f082-workload-certs\") pod \"e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8\" (UID: \"1931c057-ff82-463e-afa9-5cc33d43f082\") " pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:19.514907 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:19.514888 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1931c057-ff82-463e-afa9-5cc33d43f082-istio-envoy\") pod \"e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8\" (UID: \"1931c057-ff82-463e-afa9-5cc33d43f082\") " pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:19.515114 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:19.514909 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1931c057-ff82-463e-afa9-5cc33d43f082-istio-podinfo\") pod \"e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8\" (UID: \"1931c057-ff82-463e-afa9-5cc33d43f082\") " pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:19.615945 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:19.615905 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1931c057-ff82-463e-afa9-5cc33d43f082-workload-certs\") pod \"e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8\" (UID: \"1931c057-ff82-463e-afa9-5cc33d43f082\") " pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:19.615945 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:19.615944 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1931c057-ff82-463e-afa9-5cc33d43f082-istio-envoy\") pod \"e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8\" (UID: \"1931c057-ff82-463e-afa9-5cc33d43f082\") " pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:19.616242 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:19.615961 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1931c057-ff82-463e-afa9-5cc33d43f082-istio-podinfo\") pod \"e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8\" (UID: \"1931c057-ff82-463e-afa9-5cc33d43f082\") " pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:19.616242 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:19.615992 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1931c057-ff82-463e-afa9-5cc33d43f082-istio-data\") pod \"e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8\" (UID: \"1931c057-ff82-463e-afa9-5cc33d43f082\") " pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:19.616242 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:19.616056 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1931c057-ff82-463e-afa9-5cc33d43f082-credential-socket\") pod \"e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8\" (UID: \"1931c057-ff82-463e-afa9-5cc33d43f082\") " pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:19.616242 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:19.616106 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1931c057-ff82-463e-afa9-5cc33d43f082-workload-socket\") pod \"e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8\" (UID: \"1931c057-ff82-463e-afa9-5cc33d43f082\") " pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:19.616242 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:19.616131 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jml2d\" (UniqueName: \"kubernetes.io/projected/1931c057-ff82-463e-afa9-5cc33d43f082-kube-api-access-jml2d\") pod \"e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8\" (UID: \"1931c057-ff82-463e-afa9-5cc33d43f082\") " pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:19.616506 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:19.616247 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1931c057-ff82-463e-afa9-5cc33d43f082-istio-token\") pod \"e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8\" (UID: \"1931c057-ff82-463e-afa9-5cc33d43f082\") " pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:19.616506 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:19.616299 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1931c057-ff82-463e-afa9-5cc33d43f082-istiod-ca-cert\") pod \"e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8\" (UID: \"1931c057-ff82-463e-afa9-5cc33d43f082\") " pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:19.616506 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:19.616440 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1931c057-ff82-463e-afa9-5cc33d43f082-workload-certs\") pod \"e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8\" (UID: \"1931c057-ff82-463e-afa9-5cc33d43f082\") " pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:19.616975 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:19.616939 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1931c057-ff82-463e-afa9-5cc33d43f082-istio-data\") pod \"e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8\" (UID: \"1931c057-ff82-463e-afa9-5cc33d43f082\") " pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:19.617112 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:19.616982 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1931c057-ff82-463e-afa9-5cc33d43f082-credential-socket\") pod \"e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8\" (UID: \"1931c057-ff82-463e-afa9-5cc33d43f082\") " pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:19.617112 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:19.616992 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1931c057-ff82-463e-afa9-5cc33d43f082-workload-socket\") pod \"e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8\" (UID: \"1931c057-ff82-463e-afa9-5cc33d43f082\") " pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:19.617112 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:19.617016 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1931c057-ff82-463e-afa9-5cc33d43f082-istiod-ca-cert\") pod \"e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8\" (UID: \"1931c057-ff82-463e-afa9-5cc33d43f082\") " pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:19.618275 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:19.618255 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1931c057-ff82-463e-afa9-5cc33d43f082-istio-envoy\") pod \"e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8\" (UID: \"1931c057-ff82-463e-afa9-5cc33d43f082\") " pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:19.618518 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:19.618501 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1931c057-ff82-463e-afa9-5cc33d43f082-istio-podinfo\") pod \"e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8\" (UID: \"1931c057-ff82-463e-afa9-5cc33d43f082\") " pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:19.632390 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:19.632360 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1931c057-ff82-463e-afa9-5cc33d43f082-istio-token\") pod \"e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8\" (UID: \"1931c057-ff82-463e-afa9-5cc33d43f082\") " pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:19.634255 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:19.634236 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jml2d\" (UniqueName: \"kubernetes.io/projected/1931c057-ff82-463e-afa9-5cc33d43f082-kube-api-access-jml2d\") pod \"e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8\" (UID: \"1931c057-ff82-463e-afa9-5cc33d43f082\") " pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:19.778481 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:19.778385 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:19.922551 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:19.922528 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8"] Jul 28 01:52:19.923430 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:52:19.923408 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1931c057_ff82_463e_afa9_5cc33d43f082.slice/crio-f5785e6404226ed054de4c7e707e9da76a56f2da09d8cbf2d829a12406748538 WatchSource:0}: Error finding container f5785e6404226ed054de4c7e707e9da76a56f2da09d8cbf2d829a12406748538: Status 404 returned error can't find the container with id f5785e6404226ed054de4c7e707e9da76a56f2da09d8cbf2d829a12406748538 Jul 28 01:52:19.925454 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:19.925424 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:52:19.925524 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:19.925488 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:52:19.925524 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:19.925514 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:52:20.652950 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:20.652911 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" event={"ID":"1931c057-ff82-463e-afa9-5cc33d43f082","Type":"ContainerStarted","Data":"c0ebfdab2467c3472588121bd69672f7b6f12eb33fd0783633863e62c82c3d52"} Jul 28 01:52:20.653387 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:20.652970 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" event={"ID":"1931c057-ff82-463e-afa9-5cc33d43f082","Type":"ContainerStarted","Data":"f5785e6404226ed054de4c7e707e9da76a56f2da09d8cbf2d829a12406748538"} Jul 28 01:52:20.673348 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:20.673295 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" podStartSLOduration=1.673279193 podStartE2EDuration="1.673279193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:52:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:52:20.671584032 +0000 UTC m=+2198.105982345" watchObservedRunningTime="2026-07-28 01:52:20.673279193 +0000 UTC m=+2198.107677543" Jul 28 01:52:20.778817 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:20.778770 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:20.789393 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:20.789364 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:21.658839 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:21.658808 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:21.659811 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:21.659789 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:22.640242 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:22.640202 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4" Jul 28 01:52:22.665958 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:22.665921 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx"] Jul 28 01:52:22.673951 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:22.673890 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx" Jul 28 01:52:22.677382 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:22.677354 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-afe8f9-serving-cert\"" Jul 28 01:52:22.678934 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:22.678910 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx"] Jul 28 01:52:22.745760 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:22.745717 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bf161bd-365c-4cb0-9f36-82530b7ce5c2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx\" (UID: \"4bf161bd-365c-4cb0-9f36-82530b7ce5c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx" Jul 28 01:52:22.745951 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:22.745838 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bf161bd-365c-4cb0-9f36-82530b7ce5c2-maas-api-tls\") pod \"maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx\" (UID: \"4bf161bd-365c-4cb0-9f36-82530b7ce5c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx" Jul 28 01:52:22.746057 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:22.745952 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-988gk\" (UniqueName: \"kubernetes.io/projected/4bf161bd-365c-4cb0-9f36-82530b7ce5c2-kube-api-access-988gk\") pod \"maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx\" (UID: \"4bf161bd-365c-4cb0-9f36-82530b7ce5c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx" Jul 28 01:52:22.846926 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:22.846884 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bf161bd-365c-4cb0-9f36-82530b7ce5c2-maas-api-tls\") pod \"maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx\" (UID: \"4bf161bd-365c-4cb0-9f36-82530b7ce5c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx" Jul 28 01:52:22.847162 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:22.846953 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-988gk\" (UniqueName: \"kubernetes.io/projected/4bf161bd-365c-4cb0-9f36-82530b7ce5c2-kube-api-access-988gk\") pod \"maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx\" (UID: \"4bf161bd-365c-4cb0-9f36-82530b7ce5c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx" Jul 28 01:52:22.847162 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:22.846991 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bf161bd-365c-4cb0-9f36-82530b7ce5c2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx\" (UID: \"4bf161bd-365c-4cb0-9f36-82530b7ce5c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx" Jul 28 01:52:22.847162 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:52:22.847071 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-afe8f9-serving-cert: secret "maas-api-e2e-inf-b-afe8f9-serving-cert" not found Jul 28 01:52:22.847162 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:52:22.847144 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bf161bd-365c-4cb0-9f36-82530b7ce5c2-maas-api-tls podName:4bf161bd-365c-4cb0-9f36-82530b7ce5c2 nodeName:}" failed. No retries permitted until 2026-07-28 01:52:23.347124703 +0000 UTC m=+2200.781523015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4bf161bd-365c-4cb0-9f36-82530b7ce5c2-maas-api-tls") pod "maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx" (UID: "4bf161bd-365c-4cb0-9f36-82530b7ce5c2") : secret "maas-api-e2e-inf-b-afe8f9-serving-cert" not found Jul 28 01:52:22.847631 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:22.847611 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bf161bd-365c-4cb0-9f36-82530b7ce5c2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx\" (UID: \"4bf161bd-365c-4cb0-9f36-82530b7ce5c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx" Jul 28 01:52:22.856494 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:22.856464 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-988gk\" (UniqueName: \"kubernetes.io/projected/4bf161bd-365c-4cb0-9f36-82530b7ce5c2-kube-api-access-988gk\") pod \"maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx\" (UID: \"4bf161bd-365c-4cb0-9f36-82530b7ce5c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx" Jul 28 01:52:23.352210 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:23.352173 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bf161bd-365c-4cb0-9f36-82530b7ce5c2-maas-api-tls\") pod \"maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx\" (UID: \"4bf161bd-365c-4cb0-9f36-82530b7ce5c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx" Jul 28 01:52:23.354657 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:23.354631 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bf161bd-365c-4cb0-9f36-82530b7ce5c2-maas-api-tls\") pod \"maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx\" (UID: \"4bf161bd-365c-4cb0-9f36-82530b7ce5c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx" Jul 28 01:52:23.589891 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:23.589860 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx" Jul 28 01:52:23.720504 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:23.720479 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx"] Jul 28 01:52:24.678759 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:24.678667 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx" event={"ID":"4bf161bd-365c-4cb0-9f36-82530b7ce5c2","Type":"ContainerStarted","Data":"ebfd39015201eb941b6df4598f576e6ed06c7bef2d7dcff63b113af80d334775"} Jul 28 01:52:24.678759 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:24.678720 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx" event={"ID":"4bf161bd-365c-4cb0-9f36-82530b7ce5c2","Type":"ContainerStarted","Data":"c4c77b959e35e17ca84ea0c518ddb4d06d69dbb479eaea72609b9d4b3ee93eae"} Jul 28 01:52:24.678965 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:24.678772 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx" Jul 28 01:52:24.694705 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:24.694655 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx" podStartSLOduration=2.102645268 podStartE2EDuration="2.694637472s" totalImagesPullingTime="591.992204ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:52:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:52:24.693322693 +0000 UTC m=+2202.127721008" watchObservedRunningTime="2026-07-28 01:52:24.694637472 +0000 UTC m=+2202.129035785" Jul 28 01:52:30.689484 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:30.689450 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx" Jul 28 01:52:37.700270 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:37.700218 2565 patch_prober.go:28] interesting pod/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 28 01:52:37.700681 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:37.700317 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" podUID="1931c057-ff82-463e-afa9-5cc33d43f082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 28 01:52:38.756620 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:38.756588 2565 generic.go:356] "Generic (PLEG): container finished" podID="1931c057-ff82-463e-afa9-5cc33d43f082" containerID="c0ebfdab2467c3472588121bd69672f7b6f12eb33fd0783633863e62c82c3d52" exitCode=0 Jul 28 01:52:38.757129 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:38.756680 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" event={"ID":"1931c057-ff82-463e-afa9-5cc33d43f082","Type":"ContainerDied","Data":"c0ebfdab2467c3472588121bd69672f7b6f12eb33fd0783633863e62c82c3d52"} Jul 28 01:52:38.757129 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:38.757091 2565 scope.go:117] "RemoveContainer" containerID="c0ebfdab2467c3472588121bd69672f7b6f12eb33fd0783633863e62c82c3d52" Jul 28 01:52:38.758102 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:38.758074 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:52:38.758160 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:38.758142 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:52:38.758211 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:38.758179 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:52:39.763584 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:39.763547 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" event={"ID":"1931c057-ff82-463e-afa9-5cc33d43f082","Type":"ContainerStarted","Data":"fdac26c3242dffd214be3f2485f1000c2631cbf40aff6e918b2befb87c9cf150"} Jul 28 01:52:39.778996 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:39.778968 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:39.779273 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:39.779252 2565 patch_prober.go:28] interesting pod/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 28 01:52:39.779371 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:39.779302 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" podUID="1931c057-ff82-463e-afa9-5cc33d43f082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 28 01:52:40.779016 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:40.778982 2565 patch_prober.go:28] interesting pod/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 28 01:52:40.779443 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:40.779051 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" podUID="1931c057-ff82-463e-afa9-5cc33d43f082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 28 01:52:41.779312 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:41.779276 2565 patch_prober.go:28] interesting pod/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 28 01:52:41.779794 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:41.779333 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" podUID="1931c057-ff82-463e-afa9-5cc33d43f082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 28 01:52:42.779715 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:42.779682 2565 patch_prober.go:28] interesting pod/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 28 01:52:42.780127 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:42.779738 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" podUID="1931c057-ff82-463e-afa9-5cc33d43f082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 28 01:52:43.240633 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:43.240600 2565 scope.go:117] "RemoveContainer" containerID="42fa9a6d6100eaee5ede00e43be2331a8c309051afea4bcf34ca8ddc87f91182" Jul 28 01:52:43.782297 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:43.782266 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:43.782772 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:43.782592 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:43.783374 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:43.783348 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:52:54.479352 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:54.479306 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46"] Jul 28 01:52:54.482091 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:54.482055 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46" Jul 28 01:52:54.485819 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:54.485795 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-896d36\"/\"default-dockercfg-9rkzq\"" Jul 28 01:52:54.485933 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:54.485821 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-896d36\"/\"kube-root-ca.crt\"" Jul 28 01:52:54.485933 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:54.485831 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-896d36\"/\"openshift-service-ca.crt\"" Jul 28 01:52:54.485933 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:54.485828 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-896d36\"/\"test-model-896d36-kserve-self-signed-certs\"" Jul 28 01:52:54.491631 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:54.491610 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46"] Jul 28 01:52:54.540622 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:54.540583 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e56ea3a-6d53-46f9-9f75-674ea5bde297-model-cache\") pod \"test-model-896d36-kserve-55878dd7c7-s5t46\" (UID: \"9e56ea3a-6d53-46f9-9f75-674ea5bde297\") " pod="ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46" Jul 28 01:52:54.540622 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:54.540635 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e56ea3a-6d53-46f9-9f75-674ea5bde297-tmp-dir\") pod \"test-model-896d36-kserve-55878dd7c7-s5t46\" (UID: \"9e56ea3a-6d53-46f9-9f75-674ea5bde297\") " pod="ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46" Jul 28 01:52:54.540891 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:54.540713 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e56ea3a-6d53-46f9-9f75-674ea5bde297-home\") pod \"test-model-896d36-kserve-55878dd7c7-s5t46\" (UID: \"9e56ea3a-6d53-46f9-9f75-674ea5bde297\") " pod="ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46" Jul 28 01:52:54.540891 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:54.540818 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j2fw\" (UniqueName: \"kubernetes.io/projected/9e56ea3a-6d53-46f9-9f75-674ea5bde297-kube-api-access-9j2fw\") pod \"test-model-896d36-kserve-55878dd7c7-s5t46\" (UID: \"9e56ea3a-6d53-46f9-9f75-674ea5bde297\") " pod="ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46" Jul 28 01:52:54.540891 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:54.540857 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e56ea3a-6d53-46f9-9f75-674ea5bde297-tls-certs\") pod \"test-model-896d36-kserve-55878dd7c7-s5t46\" (UID: \"9e56ea3a-6d53-46f9-9f75-674ea5bde297\") " pod="ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46" Jul 28 01:52:54.541050 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:54.540909 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e56ea3a-6d53-46f9-9f75-674ea5bde297-dshm\") pod \"test-model-896d36-kserve-55878dd7c7-s5t46\" (UID: \"9e56ea3a-6d53-46f9-9f75-674ea5bde297\") " pod="ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46" Jul 28 01:52:54.641589 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:54.641549 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e56ea3a-6d53-46f9-9f75-674ea5bde297-model-cache\") pod \"test-model-896d36-kserve-55878dd7c7-s5t46\" (UID: \"9e56ea3a-6d53-46f9-9f75-674ea5bde297\") " pod="ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46" Jul 28 01:52:54.641774 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:54.641595 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e56ea3a-6d53-46f9-9f75-674ea5bde297-tmp-dir\") pod \"test-model-896d36-kserve-55878dd7c7-s5t46\" (UID: \"9e56ea3a-6d53-46f9-9f75-674ea5bde297\") " pod="ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46" Jul 28 01:52:54.641774 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:54.641631 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e56ea3a-6d53-46f9-9f75-674ea5bde297-home\") pod \"test-model-896d36-kserve-55878dd7c7-s5t46\" (UID: \"9e56ea3a-6d53-46f9-9f75-674ea5bde297\") " pod="ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46" Jul 28 01:52:54.641774 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:54.641675 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j2fw\" (UniqueName: \"kubernetes.io/projected/9e56ea3a-6d53-46f9-9f75-674ea5bde297-kube-api-access-9j2fw\") pod \"test-model-896d36-kserve-55878dd7c7-s5t46\" (UID: \"9e56ea3a-6d53-46f9-9f75-674ea5bde297\") " pod="ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46" Jul 28 01:52:54.641774 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:54.641695 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e56ea3a-6d53-46f9-9f75-674ea5bde297-tls-certs\") pod \"test-model-896d36-kserve-55878dd7c7-s5t46\" (UID: \"9e56ea3a-6d53-46f9-9f75-674ea5bde297\") " pod="ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46" Jul 28 01:52:54.641774 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:54.641723 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e56ea3a-6d53-46f9-9f75-674ea5bde297-dshm\") pod \"test-model-896d36-kserve-55878dd7c7-s5t46\" (UID: \"9e56ea3a-6d53-46f9-9f75-674ea5bde297\") " pod="ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46" Jul 28 01:52:54.642110 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:54.641996 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e56ea3a-6d53-46f9-9f75-674ea5bde297-model-cache\") pod \"test-model-896d36-kserve-55878dd7c7-s5t46\" (UID: \"9e56ea3a-6d53-46f9-9f75-674ea5bde297\") " pod="ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46" Jul 28 01:52:54.642110 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:54.642019 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e56ea3a-6d53-46f9-9f75-674ea5bde297-home\") pod \"test-model-896d36-kserve-55878dd7c7-s5t46\" (UID: \"9e56ea3a-6d53-46f9-9f75-674ea5bde297\") " pod="ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46" Jul 28 01:52:54.642229 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:54.642166 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e56ea3a-6d53-46f9-9f75-674ea5bde297-tmp-dir\") pod \"test-model-896d36-kserve-55878dd7c7-s5t46\" (UID: \"9e56ea3a-6d53-46f9-9f75-674ea5bde297\") " pod="ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46" Jul 28 01:52:54.644026 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:54.644005 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e56ea3a-6d53-46f9-9f75-674ea5bde297-dshm\") pod \"test-model-896d36-kserve-55878dd7c7-s5t46\" (UID: \"9e56ea3a-6d53-46f9-9f75-674ea5bde297\") " pod="ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46" Jul 28 01:52:54.644318 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:54.644301 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e56ea3a-6d53-46f9-9f75-674ea5bde297-tls-certs\") pod \"test-model-896d36-kserve-55878dd7c7-s5t46\" (UID: \"9e56ea3a-6d53-46f9-9f75-674ea5bde297\") " pod="ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46" Jul 28 01:52:54.650715 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:54.650693 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j2fw\" (UniqueName: \"kubernetes.io/projected/9e56ea3a-6d53-46f9-9f75-674ea5bde297-kube-api-access-9j2fw\") pod \"test-model-896d36-kserve-55878dd7c7-s5t46\" (UID: \"9e56ea3a-6d53-46f9-9f75-674ea5bde297\") " pod="ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46" Jul 28 01:52:54.792194 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:54.792109 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46" Jul 28 01:52:54.925530 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:54.925501 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46"] Jul 28 01:52:54.927006 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:52:54.926975 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e56ea3a_6d53_46f9_9f75_674ea5bde297.slice/crio-6d42157f1ded4f99b807af3f3ad14ffc282b1793a98faeab80a4028c587450bb WatchSource:0}: Error finding container 6d42157f1ded4f99b807af3f3ad14ffc282b1793a98faeab80a4028c587450bb: Status 404 returned error can't find the container with id 6d42157f1ded4f99b807af3f3ad14ffc282b1793a98faeab80a4028c587450bb Jul 28 01:52:55.845024 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:55.844970 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46" event={"ID":"9e56ea3a-6d53-46f9-9f75-674ea5bde297","Type":"ContainerStarted","Data":"1ecbed5f7fb63fb2ab88a878b21d23abb377b3dda72a50a6e11de5604a188462"} Jul 28 01:52:55.845024 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:55.845027 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46" event={"ID":"9e56ea3a-6d53-46f9-9f75-674ea5bde297","Type":"ContainerStarted","Data":"6d42157f1ded4f99b807af3f3ad14ffc282b1793a98faeab80a4028c587450bb"} Jul 28 01:52:55.864207 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:52:55.864153 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46" podStartSLOduration=1.8641375249999999 podStartE2EDuration="1.864137525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:52:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:52:55.86266069 +0000 UTC m=+2233.297059004" watchObservedRunningTime="2026-07-28 01:52:55.864137525 +0000 UTC m=+2233.298535839" Jul 28 01:53:04.792667 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:53:04.792614 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46" Jul 28 01:53:04.793229 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:53:04.792895 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46" Jul 28 01:53:04.805633 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:53:04.805608 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46" Jul 28 01:53:04.903913 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:53:04.903884 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46" Jul 28 01:54:17.562298 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:17.562255 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46"] Jul 28 01:54:29.191655 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:29.191618 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4"] Jul 28 01:54:29.192171 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:29.191919 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4" podUID="249144ab-463b-4e3a-80f1-4c41171f912f" containerName="maas-api" containerID="cri-o://66b24ec0a2c3a370c7f2a3e94d08479c02ce6cd2fab35968f021948d9d819540" gracePeriod=30 Jul 28 01:54:29.318161 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:29.318128 2565 generic.go:356] "Generic (PLEG): container finished" podID="249144ab-463b-4e3a-80f1-4c41171f912f" containerID="66b24ec0a2c3a370c7f2a3e94d08479c02ce6cd2fab35968f021948d9d819540" exitCode=0 Jul 28 01:54:29.318309 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:29.318199 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4" event={"ID":"249144ab-463b-4e3a-80f1-4c41171f912f","Type":"ContainerDied","Data":"66b24ec0a2c3a370c7f2a3e94d08479c02ce6cd2fab35968f021948d9d819540"} Jul 28 01:54:29.445613 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:29.445544 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4" Jul 28 01:54:29.486864 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:29.486828 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bft5\" (UniqueName: \"kubernetes.io/projected/249144ab-463b-4e3a-80f1-4c41171f912f-kube-api-access-2bft5\") pod \"249144ab-463b-4e3a-80f1-4c41171f912f\" (UID: \"249144ab-463b-4e3a-80f1-4c41171f912f\") " Jul 28 01:54:29.487083 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:29.486927 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/249144ab-463b-4e3a-80f1-4c41171f912f-openshift-service-ca\") pod \"249144ab-463b-4e3a-80f1-4c41171f912f\" (UID: \"249144ab-463b-4e3a-80f1-4c41171f912f\") " Jul 28 01:54:29.487083 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:29.486979 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/249144ab-463b-4e3a-80f1-4c41171f912f-maas-api-tls\") pod \"249144ab-463b-4e3a-80f1-4c41171f912f\" (UID: \"249144ab-463b-4e3a-80f1-4c41171f912f\") " Jul 28 01:54:29.487393 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:29.487360 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/249144ab-463b-4e3a-80f1-4c41171f912f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "249144ab-463b-4e3a-80f1-4c41171f912f" (UID: "249144ab-463b-4e3a-80f1-4c41171f912f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:54:29.489202 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:29.489177 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/249144ab-463b-4e3a-80f1-4c41171f912f-kube-api-access-2bft5" (OuterVolumeSpecName: "kube-api-access-2bft5") pod "249144ab-463b-4e3a-80f1-4c41171f912f" (UID: "249144ab-463b-4e3a-80f1-4c41171f912f"). InnerVolumeSpecName "kube-api-access-2bft5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:54:29.489303 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:29.489281 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/249144ab-463b-4e3a-80f1-4c41171f912f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "249144ab-463b-4e3a-80f1-4c41171f912f" (UID: "249144ab-463b-4e3a-80f1-4c41171f912f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:54:29.587932 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:29.587884 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/249144ab-463b-4e3a-80f1-4c41171f912f-maas-api-tls\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:54:29.587932 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:29.587917 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bft5\" (UniqueName: \"kubernetes.io/projected/249144ab-463b-4e3a-80f1-4c41171f912f-kube-api-access-2bft5\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:54:29.587932 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:29.587929 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/249144ab-463b-4e3a-80f1-4c41171f912f-openshift-service-ca\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:54:30.324389 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:30.324356 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4" Jul 28 01:54:30.324389 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:30.324375 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4" event={"ID":"249144ab-463b-4e3a-80f1-4c41171f912f","Type":"ContainerDied","Data":"a7c443af80d74c8d80c42da599e3ccdd1463a0928c3cb57181dc094869a5d469"} Jul 28 01:54:30.324899 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:30.324442 2565 scope.go:117] "RemoveContainer" containerID="66b24ec0a2c3a370c7f2a3e94d08479c02ce6cd2fab35968f021948d9d819540" Jul 28 01:54:30.346869 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:30.346834 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4"] Jul 28 01:54:30.349125 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:30.349095 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-896d36-7f9b64f5c7-zjvc4"] Jul 28 01:54:31.101290 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:31.101254 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="249144ab-463b-4e3a-80f1-4c41171f912f" path="/var/lib/kubelet/pods/249144ab-463b-4e3a-80f1-4c41171f912f/volumes" Jul 28 01:54:32.582100 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:32.582058 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46" podUID="9e56ea3a-6d53-46f9-9f75-674ea5bde297" containerName="main" containerID="cri-o://1ecbed5f7fb63fb2ab88a878b21d23abb377b3dda72a50a6e11de5604a188462" gracePeriod=45 Jul 28 01:54:32.830653 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:32.830630 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46" Jul 28 01:54:32.918579 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:32.918491 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e56ea3a-6d53-46f9-9f75-674ea5bde297-model-cache\") pod \"9e56ea3a-6d53-46f9-9f75-674ea5bde297\" (UID: \"9e56ea3a-6d53-46f9-9f75-674ea5bde297\") " Jul 28 01:54:32.918579 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:32.918530 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e56ea3a-6d53-46f9-9f75-674ea5bde297-home\") pod \"9e56ea3a-6d53-46f9-9f75-674ea5bde297\" (UID: \"9e56ea3a-6d53-46f9-9f75-674ea5bde297\") " Jul 28 01:54:32.918579 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:32.918556 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e56ea3a-6d53-46f9-9f75-674ea5bde297-tmp-dir\") pod \"9e56ea3a-6d53-46f9-9f75-674ea5bde297\" (UID: \"9e56ea3a-6d53-46f9-9f75-674ea5bde297\") " Jul 28 01:54:32.918884 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:32.918605 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e56ea3a-6d53-46f9-9f75-674ea5bde297-dshm\") pod \"9e56ea3a-6d53-46f9-9f75-674ea5bde297\" (UID: \"9e56ea3a-6d53-46f9-9f75-674ea5bde297\") " Jul 28 01:54:32.918884 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:32.918654 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9j2fw\" (UniqueName: \"kubernetes.io/projected/9e56ea3a-6d53-46f9-9f75-674ea5bde297-kube-api-access-9j2fw\") pod \"9e56ea3a-6d53-46f9-9f75-674ea5bde297\" (UID: \"9e56ea3a-6d53-46f9-9f75-674ea5bde297\") " Jul 28 01:54:32.918884 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:32.918684 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e56ea3a-6d53-46f9-9f75-674ea5bde297-tls-certs\") pod \"9e56ea3a-6d53-46f9-9f75-674ea5bde297\" (UID: \"9e56ea3a-6d53-46f9-9f75-674ea5bde297\") " Jul 28 01:54:32.918884 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:32.918827 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e56ea3a-6d53-46f9-9f75-674ea5bde297-home" (OuterVolumeSpecName: "home") pod "9e56ea3a-6d53-46f9-9f75-674ea5bde297" (UID: "9e56ea3a-6d53-46f9-9f75-674ea5bde297"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:54:32.918884 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:32.918819 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e56ea3a-6d53-46f9-9f75-674ea5bde297-model-cache" (OuterVolumeSpecName: "model-cache") pod "9e56ea3a-6d53-46f9-9f75-674ea5bde297" (UID: "9e56ea3a-6d53-46f9-9f75-674ea5bde297"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:54:32.919116 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:32.918914 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e56ea3a-6d53-46f9-9f75-674ea5bde297-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9e56ea3a-6d53-46f9-9f75-674ea5bde297" (UID: "9e56ea3a-6d53-46f9-9f75-674ea5bde297"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:54:32.919116 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:32.918931 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e56ea3a-6d53-46f9-9f75-674ea5bde297-model-cache\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:54:32.919116 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:32.918948 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e56ea3a-6d53-46f9-9f75-674ea5bde297-home\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:54:32.921006 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:32.920967 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e56ea3a-6d53-46f9-9f75-674ea5bde297-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9e56ea3a-6d53-46f9-9f75-674ea5bde297" (UID: "9e56ea3a-6d53-46f9-9f75-674ea5bde297"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:54:32.921006 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:32.920988 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e56ea3a-6d53-46f9-9f75-674ea5bde297-dshm" (OuterVolumeSpecName: "dshm") pod "9e56ea3a-6d53-46f9-9f75-674ea5bde297" (UID: "9e56ea3a-6d53-46f9-9f75-674ea5bde297"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:54:32.921173 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:32.921019 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e56ea3a-6d53-46f9-9f75-674ea5bde297-kube-api-access-9j2fw" (OuterVolumeSpecName: "kube-api-access-9j2fw") pod "9e56ea3a-6d53-46f9-9f75-674ea5bde297" (UID: "9e56ea3a-6d53-46f9-9f75-674ea5bde297"). InnerVolumeSpecName "kube-api-access-9j2fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:54:33.019401 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:33.019345 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j2fw\" (UniqueName: \"kubernetes.io/projected/9e56ea3a-6d53-46f9-9f75-674ea5bde297-kube-api-access-9j2fw\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:54:33.019401 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:33.019394 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e56ea3a-6d53-46f9-9f75-674ea5bde297-tls-certs\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:54:33.019401 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:33.019410 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e56ea3a-6d53-46f9-9f75-674ea5bde297-tmp-dir\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:54:33.019651 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:33.019421 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e56ea3a-6d53-46f9-9f75-674ea5bde297-dshm\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:54:33.347131 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:33.347088 2565 generic.go:356] "Generic (PLEG): container finished" podID="9e56ea3a-6d53-46f9-9f75-674ea5bde297" containerID="1ecbed5f7fb63fb2ab88a878b21d23abb377b3dda72a50a6e11de5604a188462" exitCode=0 Jul 28 01:54:33.347289 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:33.347155 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46" Jul 28 01:54:33.347289 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:33.347175 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46" event={"ID":"9e56ea3a-6d53-46f9-9f75-674ea5bde297","Type":"ContainerDied","Data":"1ecbed5f7fb63fb2ab88a878b21d23abb377b3dda72a50a6e11de5604a188462"} Jul 28 01:54:33.347289 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:33.347219 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46" event={"ID":"9e56ea3a-6d53-46f9-9f75-674ea5bde297","Type":"ContainerDied","Data":"6d42157f1ded4f99b807af3f3ad14ffc282b1793a98faeab80a4028c587450bb"} Jul 28 01:54:33.347289 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:33.347238 2565 scope.go:117] "RemoveContainer" containerID="1ecbed5f7fb63fb2ab88a878b21d23abb377b3dda72a50a6e11de5604a188462" Jul 28 01:54:33.355932 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:33.355910 2565 scope.go:117] "RemoveContainer" containerID="1ecbed5f7fb63fb2ab88a878b21d23abb377b3dda72a50a6e11de5604a188462" Jul 28 01:54:33.356228 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:54:33.356204 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ecbed5f7fb63fb2ab88a878b21d23abb377b3dda72a50a6e11de5604a188462\": container with ID starting with 1ecbed5f7fb63fb2ab88a878b21d23abb377b3dda72a50a6e11de5604a188462 not found: ID does not exist" containerID="1ecbed5f7fb63fb2ab88a878b21d23abb377b3dda72a50a6e11de5604a188462" Jul 28 01:54:33.356320 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:33.356248 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ecbed5f7fb63fb2ab88a878b21d23abb377b3dda72a50a6e11de5604a188462"} err="failed to get container status \"1ecbed5f7fb63fb2ab88a878b21d23abb377b3dda72a50a6e11de5604a188462\": rpc error: code = NotFound desc = could not find container \"1ecbed5f7fb63fb2ab88a878b21d23abb377b3dda72a50a6e11de5604a188462\": container with ID starting with 1ecbed5f7fb63fb2ab88a878b21d23abb377b3dda72a50a6e11de5604a188462 not found: ID does not exist" Jul 28 01:54:33.364212 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:33.364183 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46"] Jul 28 01:54:33.368170 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:33.368144 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-896d36/test-model-896d36-kserve-55878dd7c7-s5t46"] Jul 28 01:54:35.101710 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:35.101627 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e56ea3a-6d53-46f9-9f75-674ea5bde297" path="/var/lib/kubelet/pods/9e56ea3a-6d53-46f9-9f75-674ea5bde297/volumes" Jul 28 01:54:43.270911 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:43.270877 2565 scope.go:117] "RemoveContainer" containerID="a2872a91b13b5969574b8539350362e9c77c8d38051b4e965828dff087a1f5ef" Jul 28 01:54:50.864926 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:50.864890 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx"] Jul 28 01:54:50.865474 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:50.865182 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx" podUID="4bf161bd-365c-4cb0-9f36-82530b7ce5c2" containerName="maas-api" containerID="cri-o://ebfd39015201eb941b6df4598f576e6ed06c7bef2d7dcff63b113af80d334775" gracePeriod=30 Jul 28 01:54:51.320491 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:51.320465 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx" Jul 28 01:54:51.392688 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:51.392587 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bf161bd-365c-4cb0-9f36-82530b7ce5c2-openshift-service-ca\") pod \"4bf161bd-365c-4cb0-9f36-82530b7ce5c2\" (UID: \"4bf161bd-365c-4cb0-9f36-82530b7ce5c2\") " Jul 28 01:54:51.392688 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:51.392656 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bf161bd-365c-4cb0-9f36-82530b7ce5c2-maas-api-tls\") pod \"4bf161bd-365c-4cb0-9f36-82530b7ce5c2\" (UID: \"4bf161bd-365c-4cb0-9f36-82530b7ce5c2\") " Jul 28 01:54:51.392932 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:51.392775 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-988gk\" (UniqueName: \"kubernetes.io/projected/4bf161bd-365c-4cb0-9f36-82530b7ce5c2-kube-api-access-988gk\") pod \"4bf161bd-365c-4cb0-9f36-82530b7ce5c2\" (UID: \"4bf161bd-365c-4cb0-9f36-82530b7ce5c2\") " Jul 28 01:54:51.392995 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:51.392946 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bf161bd-365c-4cb0-9f36-82530b7ce5c2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4bf161bd-365c-4cb0-9f36-82530b7ce5c2" (UID: "4bf161bd-365c-4cb0-9f36-82530b7ce5c2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:54:51.394730 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:51.394695 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bf161bd-365c-4cb0-9f36-82530b7ce5c2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4bf161bd-365c-4cb0-9f36-82530b7ce5c2" (UID: "4bf161bd-365c-4cb0-9f36-82530b7ce5c2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:54:51.394840 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:51.394779 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bf161bd-365c-4cb0-9f36-82530b7ce5c2-kube-api-access-988gk" (OuterVolumeSpecName: "kube-api-access-988gk") pod "4bf161bd-365c-4cb0-9f36-82530b7ce5c2" (UID: "4bf161bd-365c-4cb0-9f36-82530b7ce5c2"). InnerVolumeSpecName "kube-api-access-988gk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:54:51.438725 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:51.438686 2565 generic.go:356] "Generic (PLEG): container finished" podID="4bf161bd-365c-4cb0-9f36-82530b7ce5c2" containerID="ebfd39015201eb941b6df4598f576e6ed06c7bef2d7dcff63b113af80d334775" exitCode=0 Jul 28 01:54:51.438897 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:51.438738 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx" Jul 28 01:54:51.438897 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:51.438740 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx" event={"ID":"4bf161bd-365c-4cb0-9f36-82530b7ce5c2","Type":"ContainerDied","Data":"ebfd39015201eb941b6df4598f576e6ed06c7bef2d7dcff63b113af80d334775"} Jul 28 01:54:51.438897 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:51.438795 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx" event={"ID":"4bf161bd-365c-4cb0-9f36-82530b7ce5c2","Type":"ContainerDied","Data":"c4c77b959e35e17ca84ea0c518ddb4d06d69dbb479eaea72609b9d4b3ee93eae"} Jul 28 01:54:51.438897 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:51.438827 2565 scope.go:117] "RemoveContainer" containerID="ebfd39015201eb941b6df4598f576e6ed06c7bef2d7dcff63b113af80d334775" Jul 28 01:54:51.448662 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:51.448639 2565 scope.go:117] "RemoveContainer" containerID="ebfd39015201eb941b6df4598f576e6ed06c7bef2d7dcff63b113af80d334775" Jul 28 01:54:51.448946 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:54:51.448921 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebfd39015201eb941b6df4598f576e6ed06c7bef2d7dcff63b113af80d334775\": container with ID starting with ebfd39015201eb941b6df4598f576e6ed06c7bef2d7dcff63b113af80d334775 not found: ID does not exist" containerID="ebfd39015201eb941b6df4598f576e6ed06c7bef2d7dcff63b113af80d334775" Jul 28 01:54:51.449018 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:51.448959 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebfd39015201eb941b6df4598f576e6ed06c7bef2d7dcff63b113af80d334775"} err="failed to get container status \"ebfd39015201eb941b6df4598f576e6ed06c7bef2d7dcff63b113af80d334775\": rpc error: code = NotFound desc = could not find container \"ebfd39015201eb941b6df4598f576e6ed06c7bef2d7dcff63b113af80d334775\": container with ID starting with ebfd39015201eb941b6df4598f576e6ed06c7bef2d7dcff63b113af80d334775 not found: ID does not exist" Jul 28 01:54:51.459653 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:51.459626 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx"] Jul 28 01:54:51.463329 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:51.463306 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-afe8f9-7c7957bdf7-pbxxx"] Jul 28 01:54:51.494205 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:51.494171 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-988gk\" (UniqueName: \"kubernetes.io/projected/4bf161bd-365c-4cb0-9f36-82530b7ce5c2-kube-api-access-988gk\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:54:51.494205 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:51.494200 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bf161bd-365c-4cb0-9f36-82530b7ce5c2-openshift-service-ca\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:54:51.494374 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:51.494216 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bf161bd-365c-4cb0-9f36-82530b7ce5c2-maas-api-tls\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:54:53.106929 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:54:53.106892 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bf161bd-365c-4cb0-9f36-82530b7ce5c2" path="/var/lib/kubelet/pods/4bf161bd-365c-4cb0-9f36-82530b7ce5c2/volumes" Jul 28 01:55:01.730006 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:01.729972 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8"] Jul 28 01:55:01.730523 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:01.730227 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" podUID="1931c057-ff82-463e-afa9-5cc33d43f082" containerName="istio-proxy" containerID="cri-o://fdac26c3242dffd214be3f2485f1000c2631cbf40aff6e918b2befb87c9cf150" gracePeriod=30 Jul 28 01:55:05.321796 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:05.321758 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1bc912-5776dd5d5b-mm69m"] Jul 28 01:55:05.322197 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:05.322111 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bf161bd-365c-4cb0-9f36-82530b7ce5c2" containerName="maas-api" Jul 28 01:55:05.322197 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:05.322123 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bf161bd-365c-4cb0-9f36-82530b7ce5c2" containerName="maas-api" Jul 28 01:55:05.322197 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:05.322133 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e56ea3a-6d53-46f9-9f75-674ea5bde297" containerName="main" Jul 28 01:55:05.322197 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:05.322140 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e56ea3a-6d53-46f9-9f75-674ea5bde297" containerName="main" Jul 28 01:55:05.322197 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:05.322148 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="249144ab-463b-4e3a-80f1-4c41171f912f" containerName="maas-api" Jul 28 01:55:05.322197 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:05.322154 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="249144ab-463b-4e3a-80f1-4c41171f912f" containerName="maas-api" Jul 28 01:55:05.322395 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:05.322212 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bf161bd-365c-4cb0-9f36-82530b7ce5c2" containerName="maas-api" Jul 28 01:55:05.322395 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:05.322221 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="249144ab-463b-4e3a-80f1-4c41171f912f" containerName="maas-api" Jul 28 01:55:05.322395 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:05.322229 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e56ea3a-6d53-46f9-9f75-674ea5bde297" containerName="main" Jul 28 01:55:05.326513 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:05.326492 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1bc912-5776dd5d5b-mm69m" Jul 28 01:55:05.329520 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:05.329489 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 01:55:05.329663 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:05.329605 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-62vvq\"" Jul 28 01:55:05.329663 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:05.329603 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-1bc912-serving-cert\"" Jul 28 01:55:05.330945 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:05.330924 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 28 01:55:05.335646 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:05.335618 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1bc912-5776dd5d5b-mm69m"] Jul 28 01:55:05.418890 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:05.418850 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/820baa96-c870-43d4-8638-2e810c2ba77b-openshift-service-ca\") pod \"maas-api-e2e-api-a-1bc912-5776dd5d5b-mm69m\" (UID: \"820baa96-c870-43d4-8638-2e810c2ba77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1bc912-5776dd5d5b-mm69m" Jul 28 01:55:05.419128 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:05.418934 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n94jf\" (UniqueName: \"kubernetes.io/projected/820baa96-c870-43d4-8638-2e810c2ba77b-kube-api-access-n94jf\") pod \"maas-api-e2e-api-a-1bc912-5776dd5d5b-mm69m\" (UID: \"820baa96-c870-43d4-8638-2e810c2ba77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1bc912-5776dd5d5b-mm69m" Jul 28 01:55:05.419128 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:05.419007 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/820baa96-c870-43d4-8638-2e810c2ba77b-maas-api-tls\") pod \"maas-api-e2e-api-a-1bc912-5776dd5d5b-mm69m\" (UID: \"820baa96-c870-43d4-8638-2e810c2ba77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1bc912-5776dd5d5b-mm69m" Jul 28 01:55:05.520027 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:05.519991 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/820baa96-c870-43d4-8638-2e810c2ba77b-openshift-service-ca\") pod \"maas-api-e2e-api-a-1bc912-5776dd5d5b-mm69m\" (UID: \"820baa96-c870-43d4-8638-2e810c2ba77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1bc912-5776dd5d5b-mm69m" Jul 28 01:55:05.520252 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:05.520110 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n94jf\" (UniqueName: \"kubernetes.io/projected/820baa96-c870-43d4-8638-2e810c2ba77b-kube-api-access-n94jf\") pod \"maas-api-e2e-api-a-1bc912-5776dd5d5b-mm69m\" (UID: \"820baa96-c870-43d4-8638-2e810c2ba77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1bc912-5776dd5d5b-mm69m" Jul 28 01:55:05.520252 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:05.520157 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/820baa96-c870-43d4-8638-2e810c2ba77b-maas-api-tls\") pod \"maas-api-e2e-api-a-1bc912-5776dd5d5b-mm69m\" (UID: \"820baa96-c870-43d4-8638-2e810c2ba77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1bc912-5776dd5d5b-mm69m" Jul 28 01:55:05.520946 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:05.520895 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/820baa96-c870-43d4-8638-2e810c2ba77b-openshift-service-ca\") pod \"maas-api-e2e-api-a-1bc912-5776dd5d5b-mm69m\" (UID: \"820baa96-c870-43d4-8638-2e810c2ba77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1bc912-5776dd5d5b-mm69m" Jul 28 01:55:05.523987 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:05.523882 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/820baa96-c870-43d4-8638-2e810c2ba77b-maas-api-tls\") pod \"maas-api-e2e-api-a-1bc912-5776dd5d5b-mm69m\" (UID: \"820baa96-c870-43d4-8638-2e810c2ba77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1bc912-5776dd5d5b-mm69m" Jul 28 01:55:05.537536 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:05.537501 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n94jf\" (UniqueName: \"kubernetes.io/projected/820baa96-c870-43d4-8638-2e810c2ba77b-kube-api-access-n94jf\") pod \"maas-api-e2e-api-a-1bc912-5776dd5d5b-mm69m\" (UID: \"820baa96-c870-43d4-8638-2e810c2ba77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1bc912-5776dd5d5b-mm69m" Jul 28 01:55:05.638584 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:05.638491 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1bc912-5776dd5d5b-mm69m" Jul 28 01:55:05.773629 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:05.773603 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1bc912-5776dd5d5b-mm69m"] Jul 28 01:55:05.775465 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:55:05.775436 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod820baa96_c870_43d4_8638_2e810c2ba77b.slice/crio-93c84904f87fe874ee7307150b436a45b866d154c38706c03587c6dfba91a9a8 WatchSource:0}: Error finding container 93c84904f87fe874ee7307150b436a45b866d154c38706c03587c6dfba91a9a8: Status 404 returned error can't find the container with id 93c84904f87fe874ee7307150b436a45b866d154c38706c03587c6dfba91a9a8 Jul 28 01:55:05.776850 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:05.776831 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 01:55:06.518308 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:06.518268 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1bc912-5776dd5d5b-mm69m" event={"ID":"820baa96-c870-43d4-8638-2e810c2ba77b","Type":"ContainerStarted","Data":"93c84904f87fe874ee7307150b436a45b866d154c38706c03587c6dfba91a9a8"} Jul 28 01:55:06.989948 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:06.989917 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:55:07.135622 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.135540 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1931c057-ff82-463e-afa9-5cc33d43f082-credential-socket\") pod \"1931c057-ff82-463e-afa9-5cc33d43f082\" (UID: \"1931c057-ff82-463e-afa9-5cc33d43f082\") " Jul 28 01:55:07.135622 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.135588 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1931c057-ff82-463e-afa9-5cc33d43f082-workload-certs\") pod \"1931c057-ff82-463e-afa9-5cc33d43f082\" (UID: \"1931c057-ff82-463e-afa9-5cc33d43f082\") " Jul 28 01:55:07.135840 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.135633 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1931c057-ff82-463e-afa9-5cc33d43f082-istiod-ca-cert\") pod \"1931c057-ff82-463e-afa9-5cc33d43f082\" (UID: \"1931c057-ff82-463e-afa9-5cc33d43f082\") " Jul 28 01:55:07.135840 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.135660 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1931c057-ff82-463e-afa9-5cc33d43f082-istio-envoy\") pod \"1931c057-ff82-463e-afa9-5cc33d43f082\" (UID: \"1931c057-ff82-463e-afa9-5cc33d43f082\") " Jul 28 01:55:07.135840 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.135686 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1931c057-ff82-463e-afa9-5cc33d43f082-istio-token\") pod \"1931c057-ff82-463e-afa9-5cc33d43f082\" (UID: \"1931c057-ff82-463e-afa9-5cc33d43f082\") " Jul 28 01:55:07.135840 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.135736 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1931c057-ff82-463e-afa9-5cc33d43f082-istio-data\") pod \"1931c057-ff82-463e-afa9-5cc33d43f082\" (UID: \"1931c057-ff82-463e-afa9-5cc33d43f082\") " Jul 28 01:55:07.135840 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.135768 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1931c057-ff82-463e-afa9-5cc33d43f082-istio-podinfo\") pod \"1931c057-ff82-463e-afa9-5cc33d43f082\" (UID: \"1931c057-ff82-463e-afa9-5cc33d43f082\") " Jul 28 01:55:07.135840 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.135816 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jml2d\" (UniqueName: \"kubernetes.io/projected/1931c057-ff82-463e-afa9-5cc33d43f082-kube-api-access-jml2d\") pod \"1931c057-ff82-463e-afa9-5cc33d43f082\" (UID: \"1931c057-ff82-463e-afa9-5cc33d43f082\") " Jul 28 01:55:07.136201 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.135854 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1931c057-ff82-463e-afa9-5cc33d43f082-workload-socket\") pod \"1931c057-ff82-463e-afa9-5cc33d43f082\" (UID: \"1931c057-ff82-463e-afa9-5cc33d43f082\") " Jul 28 01:55:07.136201 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.135857 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1931c057-ff82-463e-afa9-5cc33d43f082-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1931c057-ff82-463e-afa9-5cc33d43f082" (UID: "1931c057-ff82-463e-afa9-5cc33d43f082"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:55:07.136201 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.136018 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1931c057-ff82-463e-afa9-5cc33d43f082-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1931c057-ff82-463e-afa9-5cc33d43f082" (UID: "1931c057-ff82-463e-afa9-5cc33d43f082"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:55:07.136201 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.136140 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1931c057-ff82-463e-afa9-5cc33d43f082-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1931c057-ff82-463e-afa9-5cc33d43f082" (UID: "1931c057-ff82-463e-afa9-5cc33d43f082"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:55:07.136432 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.136289 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1931c057-ff82-463e-afa9-5cc33d43f082-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1931c057-ff82-463e-afa9-5cc33d43f082" (UID: "1931c057-ff82-463e-afa9-5cc33d43f082"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:55:07.136432 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.136348 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1931c057-ff82-463e-afa9-5cc33d43f082-credential-socket\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:55:07.136432 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.136371 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1931c057-ff82-463e-afa9-5cc33d43f082-workload-certs\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:55:07.136432 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.136387 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1931c057-ff82-463e-afa9-5cc33d43f082-istiod-ca-cert\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:55:07.137117 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.137089 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1931c057-ff82-463e-afa9-5cc33d43f082-istio-data" (OuterVolumeSpecName: "istio-data") pod "1931c057-ff82-463e-afa9-5cc33d43f082" (UID: "1931c057-ff82-463e-afa9-5cc33d43f082"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:55:07.138272 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.138233 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1931c057-ff82-463e-afa9-5cc33d43f082-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1931c057-ff82-463e-afa9-5cc33d43f082" (UID: "1931c057-ff82-463e-afa9-5cc33d43f082"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:55:07.138396 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.138372 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1931c057-ff82-463e-afa9-5cc33d43f082-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1931c057-ff82-463e-afa9-5cc33d43f082" (UID: "1931c057-ff82-463e-afa9-5cc33d43f082"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 01:55:07.138583 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.138557 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1931c057-ff82-463e-afa9-5cc33d43f082-istio-token" (OuterVolumeSpecName: "istio-token") pod "1931c057-ff82-463e-afa9-5cc33d43f082" (UID: "1931c057-ff82-463e-afa9-5cc33d43f082"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:55:07.138655 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.138566 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1931c057-ff82-463e-afa9-5cc33d43f082-kube-api-access-jml2d" (OuterVolumeSpecName: "kube-api-access-jml2d") pod "1931c057-ff82-463e-afa9-5cc33d43f082" (UID: "1931c057-ff82-463e-afa9-5cc33d43f082"). InnerVolumeSpecName "kube-api-access-jml2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:55:07.237240 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.237201 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1931c057-ff82-463e-afa9-5cc33d43f082-istio-envoy\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:55:07.237240 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.237234 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1931c057-ff82-463e-afa9-5cc33d43f082-istio-token\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:55:07.237240 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.237245 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1931c057-ff82-463e-afa9-5cc33d43f082-istio-data\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:55:07.237479 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.237256 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1931c057-ff82-463e-afa9-5cc33d43f082-istio-podinfo\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:55:07.237479 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.237268 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jml2d\" (UniqueName: \"kubernetes.io/projected/1931c057-ff82-463e-afa9-5cc33d43f082-kube-api-access-jml2d\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:55:07.237479 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.237281 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1931c057-ff82-463e-afa9-5cc33d43f082-workload-socket\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:55:07.525850 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.525742 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1bc912-5776dd5d5b-mm69m" event={"ID":"820baa96-c870-43d4-8638-2e810c2ba77b","Type":"ContainerStarted","Data":"3a47615dc5b84de276410c9f35b1d588593b159bd8bbe261690b5a95e3d54bdd"} Jul 28 01:55:07.525850 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.525806 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1bc912-5776dd5d5b-mm69m" Jul 28 01:55:07.527439 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.527411 2565 generic.go:356] "Generic (PLEG): container finished" podID="1931c057-ff82-463e-afa9-5cc33d43f082" containerID="fdac26c3242dffd214be3f2485f1000c2631cbf40aff6e918b2befb87c9cf150" exitCode=0 Jul 28 01:55:07.527559 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.527448 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" Jul 28 01:55:07.527559 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.527465 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" event={"ID":"1931c057-ff82-463e-afa9-5cc33d43f082","Type":"ContainerDied","Data":"fdac26c3242dffd214be3f2485f1000c2631cbf40aff6e918b2befb87c9cf150"} Jul 28 01:55:07.527559 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.527510 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8" event={"ID":"1931c057-ff82-463e-afa9-5cc33d43f082","Type":"ContainerDied","Data":"f5785e6404226ed054de4c7e707e9da76a56f2da09d8cbf2d829a12406748538"} Jul 28 01:55:07.527559 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.527537 2565 scope.go:117] "RemoveContainer" containerID="fdac26c3242dffd214be3f2485f1000c2631cbf40aff6e918b2befb87c9cf150" Jul 28 01:55:07.540344 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.540163 2565 scope.go:117] "RemoveContainer" containerID="c0ebfdab2467c3472588121bd69672f7b6f12eb33fd0783633863e62c82c3d52" Jul 28 01:55:07.543007 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.542960 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1bc912-5776dd5d5b-mm69m" podStartSLOduration=1.520169914 podStartE2EDuration="2.542944224s" totalImagesPullingTime="1.02277431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:55:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:55:07.542465556 +0000 UTC m=+2364.976863871" watchObservedRunningTime="2026-07-28 01:55:07.542944224 +0000 UTC m=+2364.977342539" Jul 28 01:55:07.549599 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.549580 2565 scope.go:117] "RemoveContainer" containerID="fdac26c3242dffd214be3f2485f1000c2631cbf40aff6e918b2befb87c9cf150" Jul 28 01:55:07.549898 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:55:07.549876 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdac26c3242dffd214be3f2485f1000c2631cbf40aff6e918b2befb87c9cf150\": container with ID starting with fdac26c3242dffd214be3f2485f1000c2631cbf40aff6e918b2befb87c9cf150 not found: ID does not exist" containerID="fdac26c3242dffd214be3f2485f1000c2631cbf40aff6e918b2befb87c9cf150" Jul 28 01:55:07.549967 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.549909 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdac26c3242dffd214be3f2485f1000c2631cbf40aff6e918b2befb87c9cf150"} err="failed to get container status \"fdac26c3242dffd214be3f2485f1000c2631cbf40aff6e918b2befb87c9cf150\": rpc error: code = NotFound desc = could not find container \"fdac26c3242dffd214be3f2485f1000c2631cbf40aff6e918b2befb87c9cf150\": container with ID starting with fdac26c3242dffd214be3f2485f1000c2631cbf40aff6e918b2befb87c9cf150 not found: ID does not exist" Jul 28 01:55:07.549967 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.549935 2565 scope.go:117] "RemoveContainer" containerID="c0ebfdab2467c3472588121bd69672f7b6f12eb33fd0783633863e62c82c3d52" Jul 28 01:55:07.550233 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:55:07.550211 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0ebfdab2467c3472588121bd69672f7b6f12eb33fd0783633863e62c82c3d52\": container with ID starting with c0ebfdab2467c3472588121bd69672f7b6f12eb33fd0783633863e62c82c3d52 not found: ID does not exist" containerID="c0ebfdab2467c3472588121bd69672f7b6f12eb33fd0783633863e62c82c3d52" Jul 28 01:55:07.550299 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.550243 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0ebfdab2467c3472588121bd69672f7b6f12eb33fd0783633863e62c82c3d52"} err="failed to get container status \"c0ebfdab2467c3472588121bd69672f7b6f12eb33fd0783633863e62c82c3d52\": rpc error: code = NotFound desc = could not find container \"c0ebfdab2467c3472588121bd69672f7b6f12eb33fd0783633863e62c82c3d52\": container with ID starting with c0ebfdab2467c3472588121bd69672f7b6f12eb33fd0783633863e62c82c3d52 not found: ID does not exist" Jul 28 01:55:07.556168 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.556141 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8"] Jul 28 01:55:07.559602 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:07.559578 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-afe8f9-openshift-default-64d849f9d6-vmqt8"] Jul 28 01:55:09.101328 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:09.101292 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1931c057-ff82-463e-afa9-5cc33d43f082" path="/var/lib/kubelet/pods/1931c057-ff82-463e-afa9-5cc33d43f082/volumes" Jul 28 01:55:13.337647 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:13.337614 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2"] Jul 28 01:55:13.338142 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:13.337958 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1931c057-ff82-463e-afa9-5cc33d43f082" containerName="istio-proxy" Jul 28 01:55:13.338142 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:13.337975 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1931c057-ff82-463e-afa9-5cc33d43f082" containerName="istio-proxy" Jul 28 01:55:13.338142 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:13.338099 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1931c057-ff82-463e-afa9-5cc33d43f082" containerName="istio-proxy" Jul 28 01:55:13.342570 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:13.342545 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2" Jul 28 01:55:13.344913 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:13.344887 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-4b0f72-serving-cert\"" Jul 28 01:55:13.353430 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:13.353403 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2"] Jul 28 01:55:13.390060 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:13.390000 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b2475cf-1417-49e6-867b-f49f62622430-openshift-service-ca\") pod \"maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2\" (UID: \"1b2475cf-1417-49e6-867b-f49f62622430\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2" Jul 28 01:55:13.390060 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:13.390058 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbhvr\" (UniqueName: \"kubernetes.io/projected/1b2475cf-1417-49e6-867b-f49f62622430-kube-api-access-bbhvr\") pod \"maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2\" (UID: \"1b2475cf-1417-49e6-867b-f49f62622430\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2" Jul 28 01:55:13.390273 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:13.390111 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b2475cf-1417-49e6-867b-f49f62622430-maas-api-tls\") pod \"maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2\" (UID: \"1b2475cf-1417-49e6-867b-f49f62622430\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2" Jul 28 01:55:13.490729 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:13.490684 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b2475cf-1417-49e6-867b-f49f62622430-openshift-service-ca\") pod \"maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2\" (UID: \"1b2475cf-1417-49e6-867b-f49f62622430\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2" Jul 28 01:55:13.490926 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:13.490747 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbhvr\" (UniqueName: \"kubernetes.io/projected/1b2475cf-1417-49e6-867b-f49f62622430-kube-api-access-bbhvr\") pod \"maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2\" (UID: \"1b2475cf-1417-49e6-867b-f49f62622430\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2" Jul 28 01:55:13.490926 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:13.490796 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b2475cf-1417-49e6-867b-f49f62622430-maas-api-tls\") pod \"maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2\" (UID: \"1b2475cf-1417-49e6-867b-f49f62622430\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2" Jul 28 01:55:13.491074 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:55:13.490928 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-4b0f72-serving-cert: secret "maas-api-e2e-api-b-4b0f72-serving-cert" not found Jul 28 01:55:13.491074 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:55:13.491009 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b2475cf-1417-49e6-867b-f49f62622430-maas-api-tls podName:1b2475cf-1417-49e6-867b-f49f62622430 nodeName:}" failed. No retries permitted until 2026-07-28 01:55:13.990987901 +0000 UTC m=+2371.425386217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1b2475cf-1417-49e6-867b-f49f62622430-maas-api-tls") pod "maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2" (UID: "1b2475cf-1417-49e6-867b-f49f62622430") : secret "maas-api-e2e-api-b-4b0f72-serving-cert" not found Jul 28 01:55:13.491442 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:13.491420 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b2475cf-1417-49e6-867b-f49f62622430-openshift-service-ca\") pod \"maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2\" (UID: \"1b2475cf-1417-49e6-867b-f49f62622430\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2" Jul 28 01:55:13.500549 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:13.500498 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbhvr\" (UniqueName: \"kubernetes.io/projected/1b2475cf-1417-49e6-867b-f49f62622430-kube-api-access-bbhvr\") pod \"maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2\" (UID: \"1b2475cf-1417-49e6-867b-f49f62622430\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2" Jul 28 01:55:13.539051 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:13.539007 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1bc912-5776dd5d5b-mm69m" Jul 28 01:55:13.994684 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:13.994650 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b2475cf-1417-49e6-867b-f49f62622430-maas-api-tls\") pod \"maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2\" (UID: \"1b2475cf-1417-49e6-867b-f49f62622430\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2" Jul 28 01:55:13.997105 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:13.997079 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b2475cf-1417-49e6-867b-f49f62622430-maas-api-tls\") pod \"maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2\" (UID: \"1b2475cf-1417-49e6-867b-f49f62622430\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2" Jul 28 01:55:14.256670 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:14.256515 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2" Jul 28 01:55:14.397472 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:14.397444 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2"] Jul 28 01:55:14.398952 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:55:14.398925 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b2475cf_1417_49e6_867b_f49f62622430.slice/crio-367e864a216b740df8b6bea5c8857fbbbf682cecd2d1d79505e879ada0814dc8 WatchSource:0}: Error finding container 367e864a216b740df8b6bea5c8857fbbbf682cecd2d1d79505e879ada0814dc8: Status 404 returned error can't find the container with id 367e864a216b740df8b6bea5c8857fbbbf682cecd2d1d79505e879ada0814dc8 Jul 28 01:55:14.566073 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:14.565952 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2" event={"ID":"1b2475cf-1417-49e6-867b-f49f62622430","Type":"ContainerStarted","Data":"367e864a216b740df8b6bea5c8857fbbbf682cecd2d1d79505e879ada0814dc8"} Jul 28 01:55:15.572114 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:15.572077 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2" event={"ID":"1b2475cf-1417-49e6-867b-f49f62622430","Type":"ContainerStarted","Data":"f7ec0de1dfad33ad29afa9bef1c6c5e2e2cd3c59cd0073961dca071cba618ac9"} Jul 28 01:55:15.572511 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:15.572159 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2" Jul 28 01:55:15.588500 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:15.588451 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2" podStartSLOduration=2.268900303 podStartE2EDuration="2.588435513s" totalImagesPullingTime="319.53521ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:55:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:55:15.586810209 +0000 UTC m=+2373.021208722" watchObservedRunningTime="2026-07-28 01:55:15.588435513 +0000 UTC m=+2373.022833827" Jul 28 01:55:21.582466 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:21.582430 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2" Jul 28 01:55:25.828044 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:25.827991 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1bc912-45w2l"] Jul 28 01:55:25.828539 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:25.828468 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1931c057-ff82-463e-afa9-5cc33d43f082" containerName="istio-proxy" Jul 28 01:55:25.828539 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:25.828486 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1931c057-ff82-463e-afa9-5cc33d43f082" containerName="istio-proxy" Jul 28 01:55:25.828660 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:25.828565 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1931c057-ff82-463e-afa9-5cc33d43f082" containerName="istio-proxy" Jul 28 01:55:25.831637 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:25.831617 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1bc912-45w2l" Jul 28 01:55:25.834129 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:25.834102 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wjxvt\"" Jul 28 01:55:25.838514 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:25.838456 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1bc912-45w2l"] Jul 28 01:55:25.900432 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:25.900387 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f43afb51-0589-442c-b14a-28abc271870e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1bc912-45w2l\" (UID: \"f43afb51-0589-442c-b14a-28abc271870e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1bc912-45w2l" Jul 28 01:55:26.001270 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:26.001224 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f43afb51-0589-442c-b14a-28abc271870e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1bc912-45w2l\" (UID: \"f43afb51-0589-442c-b14a-28abc271870e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1bc912-45w2l" Jul 28 01:55:26.001883 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:26.001861 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f43afb51-0589-442c-b14a-28abc271870e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1bc912-45w2l\" (UID: \"f43afb51-0589-442c-b14a-28abc271870e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1bc912-45w2l" Jul 28 01:55:26.144189 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:26.144094 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1bc912-45w2l" Jul 28 01:55:26.276432 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:26.276403 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1bc912-45w2l"] Jul 28 01:55:26.280237 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:55:26.280199 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43afb51_0589_442c_b14a_28abc271870e.slice/crio-667a2dbe9b9690355418736a50d8bd5499581dfc68ce2f94484e43f1d5292264 WatchSource:0}: Error finding container 667a2dbe9b9690355418736a50d8bd5499581dfc68ce2f94484e43f1d5292264: Status 404 returned error can't find the container with id 667a2dbe9b9690355418736a50d8bd5499581dfc68ce2f94484e43f1d5292264 Jul 28 01:55:26.633823 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:26.628452 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1bc912-45w2l" event={"ID":"f43afb51-0589-442c-b14a-28abc271870e","Type":"ContainerStarted","Data":"ed0f66498da0d6b78d574fab1d18e395182fac0c18ddf398646e8baeab6899cf"} Jul 28 01:55:26.633823 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:26.628598 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1bc912-45w2l" event={"ID":"f43afb51-0589-442c-b14a-28abc271870e","Type":"ContainerStarted","Data":"667a2dbe9b9690355418736a50d8bd5499581dfc68ce2f94484e43f1d5292264"} Jul 28 01:55:26.647672 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:26.647610 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1bc912-45w2l" podStartSLOduration=1.647593861 podStartE2EDuration="1.647593861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:55:26.645465879 +0000 UTC m=+2384.079864187" watchObservedRunningTime="2026-07-28 01:55:26.647593861 +0000 UTC m=+2384.081992176" Jul 28 01:55:27.634582 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:27.634544 2565 generic.go:356] "Generic (PLEG): container finished" podID="f43afb51-0589-442c-b14a-28abc271870e" containerID="ed0f66498da0d6b78d574fab1d18e395182fac0c18ddf398646e8baeab6899cf" exitCode=0 Jul 28 01:55:27.634957 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:27.634603 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1bc912-45w2l" event={"ID":"f43afb51-0589-442c-b14a-28abc271870e","Type":"ContainerDied","Data":"ed0f66498da0d6b78d574fab1d18e395182fac0c18ddf398646e8baeab6899cf"} Jul 28 01:55:28.768002 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:28.767973 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1bc912-45w2l" Jul 28 01:55:28.824496 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:28.824467 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f43afb51-0589-442c-b14a-28abc271870e-maas-api-service-ca\") pod \"f43afb51-0589-442c-b14a-28abc271870e\" (UID: \"f43afb51-0589-442c-b14a-28abc271870e\") " Jul 28 01:55:28.824815 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:28.824792 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f43afb51-0589-442c-b14a-28abc271870e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f43afb51-0589-442c-b14a-28abc271870e" (UID: "f43afb51-0589-442c-b14a-28abc271870e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:55:28.926017 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:28.925912 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f43afb51-0589-442c-b14a-28abc271870e-maas-api-service-ca\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:55:29.645831 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:29.645800 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1bc912-45w2l" event={"ID":"f43afb51-0589-442c-b14a-28abc271870e","Type":"ContainerDied","Data":"667a2dbe9b9690355418736a50d8bd5499581dfc68ce2f94484e43f1d5292264"} Jul 28 01:55:29.645831 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:29.645823 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1bc912-45w2l" Jul 28 01:55:29.646068 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:29.645846 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="667a2dbe9b9690355418736a50d8bd5499581dfc68ce2f94484e43f1d5292264" Jul 28 01:55:40.912830 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:40.912782 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1bc912-5776dd5d5b-mm69m"] Jul 28 01:55:40.913315 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:40.913119 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1bc912-5776dd5d5b-mm69m" podUID="820baa96-c870-43d4-8638-2e810c2ba77b" containerName="maas-api" containerID="cri-o://3a47615dc5b84de276410c9f35b1d588593b159bd8bbe261690b5a95e3d54bdd" gracePeriod=30 Jul 28 01:55:41.166152 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:41.166086 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1bc912-5776dd5d5b-mm69m" Jul 28 01:55:41.235357 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:41.235318 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/820baa96-c870-43d4-8638-2e810c2ba77b-maas-api-tls\") pod \"820baa96-c870-43d4-8638-2e810c2ba77b\" (UID: \"820baa96-c870-43d4-8638-2e810c2ba77b\") " Jul 28 01:55:41.235536 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:41.235368 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/820baa96-c870-43d4-8638-2e810c2ba77b-openshift-service-ca\") pod \"820baa96-c870-43d4-8638-2e810c2ba77b\" (UID: \"820baa96-c870-43d4-8638-2e810c2ba77b\") " Jul 28 01:55:41.235536 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:41.235396 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n94jf\" (UniqueName: \"kubernetes.io/projected/820baa96-c870-43d4-8638-2e810c2ba77b-kube-api-access-n94jf\") pod \"820baa96-c870-43d4-8638-2e810c2ba77b\" (UID: \"820baa96-c870-43d4-8638-2e810c2ba77b\") " Jul 28 01:55:41.235814 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:41.235790 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/820baa96-c870-43d4-8638-2e810c2ba77b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "820baa96-c870-43d4-8638-2e810c2ba77b" (UID: "820baa96-c870-43d4-8638-2e810c2ba77b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:55:41.237679 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:41.237649 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/820baa96-c870-43d4-8638-2e810c2ba77b-kube-api-access-n94jf" (OuterVolumeSpecName: "kube-api-access-n94jf") pod "820baa96-c870-43d4-8638-2e810c2ba77b" (UID: "820baa96-c870-43d4-8638-2e810c2ba77b"). InnerVolumeSpecName "kube-api-access-n94jf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:55:41.237679 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:41.237650 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/820baa96-c870-43d4-8638-2e810c2ba77b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "820baa96-c870-43d4-8638-2e810c2ba77b" (UID: "820baa96-c870-43d4-8638-2e810c2ba77b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:55:41.336471 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:41.336428 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/820baa96-c870-43d4-8638-2e810c2ba77b-maas-api-tls\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:55:41.336471 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:41.336465 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/820baa96-c870-43d4-8638-2e810c2ba77b-openshift-service-ca\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:55:41.336471 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:41.336480 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n94jf\" (UniqueName: \"kubernetes.io/projected/820baa96-c870-43d4-8638-2e810c2ba77b-kube-api-access-n94jf\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:55:41.709179 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:41.709147 2565 generic.go:356] "Generic (PLEG): container finished" podID="820baa96-c870-43d4-8638-2e810c2ba77b" containerID="3a47615dc5b84de276410c9f35b1d588593b159bd8bbe261690b5a95e3d54bdd" exitCode=0 Jul 28 01:55:41.709366 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:41.709196 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1bc912-5776dd5d5b-mm69m" Jul 28 01:55:41.709366 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:41.709239 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1bc912-5776dd5d5b-mm69m" event={"ID":"820baa96-c870-43d4-8638-2e810c2ba77b","Type":"ContainerDied","Data":"3a47615dc5b84de276410c9f35b1d588593b159bd8bbe261690b5a95e3d54bdd"} Jul 28 01:55:41.709366 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:41.709299 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1bc912-5776dd5d5b-mm69m" event={"ID":"820baa96-c870-43d4-8638-2e810c2ba77b","Type":"ContainerDied","Data":"93c84904f87fe874ee7307150b436a45b866d154c38706c03587c6dfba91a9a8"} Jul 28 01:55:41.709366 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:41.709329 2565 scope.go:117] "RemoveContainer" containerID="3a47615dc5b84de276410c9f35b1d588593b159bd8bbe261690b5a95e3d54bdd" Jul 28 01:55:41.718404 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:41.718382 2565 scope.go:117] "RemoveContainer" containerID="3a47615dc5b84de276410c9f35b1d588593b159bd8bbe261690b5a95e3d54bdd" Jul 28 01:55:41.718726 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:55:41.718707 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a47615dc5b84de276410c9f35b1d588593b159bd8bbe261690b5a95e3d54bdd\": container with ID starting with 3a47615dc5b84de276410c9f35b1d588593b159bd8bbe261690b5a95e3d54bdd not found: ID does not exist" containerID="3a47615dc5b84de276410c9f35b1d588593b159bd8bbe261690b5a95e3d54bdd" Jul 28 01:55:41.718778 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:41.718738 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a47615dc5b84de276410c9f35b1d588593b159bd8bbe261690b5a95e3d54bdd"} err="failed to get container status \"3a47615dc5b84de276410c9f35b1d588593b159bd8bbe261690b5a95e3d54bdd\": rpc error: code = NotFound desc = could not find container \"3a47615dc5b84de276410c9f35b1d588593b159bd8bbe261690b5a95e3d54bdd\": container with ID starting with 3a47615dc5b84de276410c9f35b1d588593b159bd8bbe261690b5a95e3d54bdd not found: ID does not exist" Jul 28 01:55:41.730279 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:41.730251 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1bc912-5776dd5d5b-mm69m"] Jul 28 01:55:41.733922 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:41.733899 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1bc912-5776dd5d5b-mm69m"] Jul 28 01:55:43.101395 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:43.101360 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="820baa96-c870-43d4-8638-2e810c2ba77b" path="/var/lib/kubelet/pods/820baa96-c870-43d4-8638-2e810c2ba77b/volumes" Jul 28 01:55:45.914704 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:45.914670 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1bc912-45w2l"] Jul 28 01:55:45.917966 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:45.917942 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1bc912-45w2l"] Jul 28 01:55:47.101138 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:47.101100 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f43afb51-0589-442c-b14a-28abc271870e" path="/var/lib/kubelet/pods/f43afb51-0589-442c-b14a-28abc271870e/volumes" Jul 28 01:55:52.460170 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:52.460123 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b0f72-zf6bq"] Jul 28 01:55:52.460693 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:52.460450 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="820baa96-c870-43d4-8638-2e810c2ba77b" containerName="maas-api" Jul 28 01:55:52.460693 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:52.460460 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="820baa96-c870-43d4-8638-2e810c2ba77b" containerName="maas-api" Jul 28 01:55:52.460693 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:52.460474 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f43afb51-0589-442c-b14a-28abc271870e" containerName="revoke-keys" Jul 28 01:55:52.460693 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:52.460479 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f43afb51-0589-442c-b14a-28abc271870e" containerName="revoke-keys" Jul 28 01:55:52.460693 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:52.460525 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f43afb51-0589-442c-b14a-28abc271870e" containerName="revoke-keys" Jul 28 01:55:52.460693 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:52.460536 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="820baa96-c870-43d4-8638-2e810c2ba77b" containerName="maas-api" Jul 28 01:55:52.463305 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:52.463288 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b0f72-zf6bq" Jul 28 01:55:52.465785 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:52.465764 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wjxvt\"" Jul 28 01:55:52.470191 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:52.470156 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b0f72-zf6bq"] Jul 28 01:55:52.640102 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:52.640059 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b3ff05-0102-4fc1-8c5f-a9d433f03eff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4b0f72-zf6bq\" (UID: \"40b3ff05-0102-4fc1-8c5f-a9d433f03eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b0f72-zf6bq" Jul 28 01:55:52.741591 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:52.741501 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b3ff05-0102-4fc1-8c5f-a9d433f03eff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4b0f72-zf6bq\" (UID: \"40b3ff05-0102-4fc1-8c5f-a9d433f03eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b0f72-zf6bq" Jul 28 01:55:52.742164 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:52.742141 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b3ff05-0102-4fc1-8c5f-a9d433f03eff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4b0f72-zf6bq\" (UID: \"40b3ff05-0102-4fc1-8c5f-a9d433f03eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b0f72-zf6bq" Jul 28 01:55:52.775137 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:52.775106 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b0f72-zf6bq" Jul 28 01:55:52.908119 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:52.908084 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b0f72-zf6bq"] Jul 28 01:55:52.909542 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:55:52.909520 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b3ff05_0102_4fc1_8c5f_a9d433f03eff.slice/crio-44389d48f7d1f11bceb47ed76040d80b6182fa9ceaec5af2b2c324b662bbfd95 WatchSource:0}: Error finding container 44389d48f7d1f11bceb47ed76040d80b6182fa9ceaec5af2b2c324b662bbfd95: Status 404 returned error can't find the container with id 44389d48f7d1f11bceb47ed76040d80b6182fa9ceaec5af2b2c324b662bbfd95 Jul 28 01:55:53.771088 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:53.770978 2565 generic.go:356] "Generic (PLEG): container finished" podID="40b3ff05-0102-4fc1-8c5f-a9d433f03eff" containerID="94ec1b91e6a07224e685efe61f39eac510b413d690991f9edfb1b1790351b51d" exitCode=0 Jul 28 01:55:53.771088 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:53.771061 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b0f72-zf6bq" event={"ID":"40b3ff05-0102-4fc1-8c5f-a9d433f03eff","Type":"ContainerDied","Data":"94ec1b91e6a07224e685efe61f39eac510b413d690991f9edfb1b1790351b51d"} Jul 28 01:55:53.771479 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:53.771115 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b0f72-zf6bq" event={"ID":"40b3ff05-0102-4fc1-8c5f-a9d433f03eff","Type":"ContainerStarted","Data":"44389d48f7d1f11bceb47ed76040d80b6182fa9ceaec5af2b2c324b662bbfd95"} Jul 28 01:55:54.906212 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:54.906184 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b0f72-zf6bq" Jul 28 01:55:55.065336 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:55.065244 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b3ff05-0102-4fc1-8c5f-a9d433f03eff-maas-api-service-ca\") pod \"40b3ff05-0102-4fc1-8c5f-a9d433f03eff\" (UID: \"40b3ff05-0102-4fc1-8c5f-a9d433f03eff\") " Jul 28 01:55:55.065633 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:55.065606 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40b3ff05-0102-4fc1-8c5f-a9d433f03eff-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "40b3ff05-0102-4fc1-8c5f-a9d433f03eff" (UID: "40b3ff05-0102-4fc1-8c5f-a9d433f03eff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:55:55.166017 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:55.165978 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b3ff05-0102-4fc1-8c5f-a9d433f03eff-maas-api-service-ca\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:55:55.782027 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:55.781985 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b0f72-zf6bq" event={"ID":"40b3ff05-0102-4fc1-8c5f-a9d433f03eff","Type":"ContainerDied","Data":"44389d48f7d1f11bceb47ed76040d80b6182fa9ceaec5af2b2c324b662bbfd95"} Jul 28 01:55:55.782216 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:55.782025 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b0f72-zf6bq" Jul 28 01:55:55.782216 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:55:55.782068 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44389d48f7d1f11bceb47ed76040d80b6182fa9ceaec5af2b2c324b662bbfd95" Jul 28 01:56:08.113498 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:08.113423 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2"] Jul 28 01:56:08.114001 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:08.113619 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2" podUID="1b2475cf-1417-49e6-867b-f49f62622430" containerName="maas-api" containerID="cri-o://f7ec0de1dfad33ad29afa9bef1c6c5e2e2cd3c59cd0073961dca071cba618ac9" gracePeriod=30 Jul 28 01:56:08.141525 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:56:08.141493 2565 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b2475cf_1417_49e6_867b_f49f62622430.slice/crio-f7ec0de1dfad33ad29afa9bef1c6c5e2e2cd3c59cd0073961dca071cba618ac9.scope\": RecentStats: unable to find data in memory cache]" Jul 28 01:56:08.363155 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:08.363126 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2" Jul 28 01:56:08.376958 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:08.376894 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbhvr\" (UniqueName: \"kubernetes.io/projected/1b2475cf-1417-49e6-867b-f49f62622430-kube-api-access-bbhvr\") pod \"1b2475cf-1417-49e6-867b-f49f62622430\" (UID: \"1b2475cf-1417-49e6-867b-f49f62622430\") " Jul 28 01:56:08.376958 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:08.376927 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b2475cf-1417-49e6-867b-f49f62622430-maas-api-tls\") pod \"1b2475cf-1417-49e6-867b-f49f62622430\" (UID: \"1b2475cf-1417-49e6-867b-f49f62622430\") " Jul 28 01:56:08.377209 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:08.376966 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b2475cf-1417-49e6-867b-f49f62622430-openshift-service-ca\") pod \"1b2475cf-1417-49e6-867b-f49f62622430\" (UID: \"1b2475cf-1417-49e6-867b-f49f62622430\") " Jul 28 01:56:08.377360 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:08.377329 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b2475cf-1417-49e6-867b-f49f62622430-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1b2475cf-1417-49e6-867b-f49f62622430" (UID: "1b2475cf-1417-49e6-867b-f49f62622430"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:56:08.379085 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:08.379061 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b2475cf-1417-49e6-867b-f49f62622430-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1b2475cf-1417-49e6-867b-f49f62622430" (UID: "1b2475cf-1417-49e6-867b-f49f62622430"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:56:08.379182 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:08.379100 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b2475cf-1417-49e6-867b-f49f62622430-kube-api-access-bbhvr" (OuterVolumeSpecName: "kube-api-access-bbhvr") pod "1b2475cf-1417-49e6-867b-f49f62622430" (UID: "1b2475cf-1417-49e6-867b-f49f62622430"). InnerVolumeSpecName "kube-api-access-bbhvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:56:08.478403 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:08.478373 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbhvr\" (UniqueName: \"kubernetes.io/projected/1b2475cf-1417-49e6-867b-f49f62622430-kube-api-access-bbhvr\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:56:08.478403 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:08.478403 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b2475cf-1417-49e6-867b-f49f62622430-maas-api-tls\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:56:08.478597 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:08.478417 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b2475cf-1417-49e6-867b-f49f62622430-openshift-service-ca\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 01:56:08.848422 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:08.848392 2565 generic.go:356] "Generic (PLEG): container finished" podID="1b2475cf-1417-49e6-867b-f49f62622430" containerID="f7ec0de1dfad33ad29afa9bef1c6c5e2e2cd3c59cd0073961dca071cba618ac9" exitCode=0 Jul 28 01:56:08.848604 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:08.848434 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2" Jul 28 01:56:08.848604 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:08.848482 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2" event={"ID":"1b2475cf-1417-49e6-867b-f49f62622430","Type":"ContainerDied","Data":"f7ec0de1dfad33ad29afa9bef1c6c5e2e2cd3c59cd0073961dca071cba618ac9"} Jul 28 01:56:08.848604 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:08.848534 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2" event={"ID":"1b2475cf-1417-49e6-867b-f49f62622430","Type":"ContainerDied","Data":"367e864a216b740df8b6bea5c8857fbbbf682cecd2d1d79505e879ada0814dc8"} Jul 28 01:56:08.848604 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:08.848565 2565 scope.go:117] "RemoveContainer" containerID="f7ec0de1dfad33ad29afa9bef1c6c5e2e2cd3c59cd0073961dca071cba618ac9" Jul 28 01:56:08.857584 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:08.857564 2565 scope.go:117] "RemoveContainer" containerID="f7ec0de1dfad33ad29afa9bef1c6c5e2e2cd3c59cd0073961dca071cba618ac9" Jul 28 01:56:08.857819 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:56:08.857798 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7ec0de1dfad33ad29afa9bef1c6c5e2e2cd3c59cd0073961dca071cba618ac9\": container with ID starting with f7ec0de1dfad33ad29afa9bef1c6c5e2e2cd3c59cd0073961dca071cba618ac9 not found: ID does not exist" containerID="f7ec0de1dfad33ad29afa9bef1c6c5e2e2cd3c59cd0073961dca071cba618ac9" Jul 28 01:56:08.857883 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:08.857830 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7ec0de1dfad33ad29afa9bef1c6c5e2e2cd3c59cd0073961dca071cba618ac9"} err="failed to get container status \"f7ec0de1dfad33ad29afa9bef1c6c5e2e2cd3c59cd0073961dca071cba618ac9\": rpc error: code = NotFound desc = could not find container \"f7ec0de1dfad33ad29afa9bef1c6c5e2e2cd3c59cd0073961dca071cba618ac9\": container with ID starting with f7ec0de1dfad33ad29afa9bef1c6c5e2e2cd3c59cd0073961dca071cba618ac9 not found: ID does not exist" Jul 28 01:56:08.870405 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:08.870381 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2"] Jul 28 01:56:08.875962 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:08.875943 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4b0f72-9cc58d6db-pm7d2"] Jul 28 01:56:09.100975 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:09.100895 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b2475cf-1417-49e6-867b-f49f62622430" path="/var/lib/kubelet/pods/1b2475cf-1417-49e6-867b-f49f62622430/volumes" Jul 28 01:56:12.538636 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:12.538600 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b0f72-zf6bq"] Jul 28 01:56:12.541359 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:12.541337 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b0f72-zf6bq"] Jul 28 01:56:13.100672 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:13.100638 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40b3ff05-0102-4fc1-8c5f-a9d433f03eff" path="/var/lib/kubelet/pods/40b3ff05-0102-4fc1-8c5f-a9d433f03eff/volumes" Jul 28 01:56:22.789143 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:22.789106 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-dfed9b-766788bf-md4fq"] Jul 28 01:56:22.789607 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:22.789447 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b2475cf-1417-49e6-867b-f49f62622430" containerName="maas-api" Jul 28 01:56:22.789607 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:22.789459 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b2475cf-1417-49e6-867b-f49f62622430" containerName="maas-api" Jul 28 01:56:22.789607 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:22.789469 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40b3ff05-0102-4fc1-8c5f-a9d433f03eff" containerName="revoke-keys" Jul 28 01:56:22.789607 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:22.789475 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="40b3ff05-0102-4fc1-8c5f-a9d433f03eff" containerName="revoke-keys" Jul 28 01:56:22.789607 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:22.789548 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="40b3ff05-0102-4fc1-8c5f-a9d433f03eff" containerName="revoke-keys" Jul 28 01:56:22.789607 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:22.789557 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b2475cf-1417-49e6-867b-f49f62622430" containerName="maas-api" Jul 28 01:56:22.792585 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:22.792564 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dfed9b-766788bf-md4fq" Jul 28 01:56:22.795283 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:22.795255 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-dfed9b-serving-cert\"" Jul 28 01:56:22.795283 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:22.795272 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-62vvq\"" Jul 28 01:56:22.796455 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:22.796437 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 01:56:22.796547 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:22.796461 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 28 01:56:22.802478 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:22.802455 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-dfed9b-766788bf-md4fq"] Jul 28 01:56:22.900884 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:22.900847 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82pjf\" (UniqueName: \"kubernetes.io/projected/c1b0ca2e-075a-40e7-b328-34d54d43b6d8-kube-api-access-82pjf\") pod \"maas-api-e2e-shared-a-dfed9b-766788bf-md4fq\" (UID: \"c1b0ca2e-075a-40e7-b328-34d54d43b6d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dfed9b-766788bf-md4fq" Jul 28 01:56:22.900884 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:22.900882 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1b0ca2e-075a-40e7-b328-34d54d43b6d8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-dfed9b-766788bf-md4fq\" (UID: \"c1b0ca2e-075a-40e7-b328-34d54d43b6d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dfed9b-766788bf-md4fq" Jul 28 01:56:22.901118 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:22.900969 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1b0ca2e-075a-40e7-b328-34d54d43b6d8-maas-api-tls\") pod \"maas-api-e2e-shared-a-dfed9b-766788bf-md4fq\" (UID: \"c1b0ca2e-075a-40e7-b328-34d54d43b6d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dfed9b-766788bf-md4fq" Jul 28 01:56:23.001913 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:23.001870 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82pjf\" (UniqueName: \"kubernetes.io/projected/c1b0ca2e-075a-40e7-b328-34d54d43b6d8-kube-api-access-82pjf\") pod \"maas-api-e2e-shared-a-dfed9b-766788bf-md4fq\" (UID: \"c1b0ca2e-075a-40e7-b328-34d54d43b6d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dfed9b-766788bf-md4fq" Jul 28 01:56:23.001913 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:23.001914 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1b0ca2e-075a-40e7-b328-34d54d43b6d8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-dfed9b-766788bf-md4fq\" (UID: \"c1b0ca2e-075a-40e7-b328-34d54d43b6d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dfed9b-766788bf-md4fq" Jul 28 01:56:23.002226 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:23.001981 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1b0ca2e-075a-40e7-b328-34d54d43b6d8-maas-api-tls\") pod \"maas-api-e2e-shared-a-dfed9b-766788bf-md4fq\" (UID: \"c1b0ca2e-075a-40e7-b328-34d54d43b6d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dfed9b-766788bf-md4fq" Jul 28 01:56:23.002226 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:56:23.002206 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-dfed9b-serving-cert: secret "maas-api-e2e-shared-a-dfed9b-serving-cert" not found Jul 28 01:56:23.002316 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:56:23.002299 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1b0ca2e-075a-40e7-b328-34d54d43b6d8-maas-api-tls podName:c1b0ca2e-075a-40e7-b328-34d54d43b6d8 nodeName:}" failed. No retries permitted until 2026-07-28 01:56:23.502274852 +0000 UTC m=+2440.936673144 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c1b0ca2e-075a-40e7-b328-34d54d43b6d8-maas-api-tls") pod "maas-api-e2e-shared-a-dfed9b-766788bf-md4fq" (UID: "c1b0ca2e-075a-40e7-b328-34d54d43b6d8") : secret "maas-api-e2e-shared-a-dfed9b-serving-cert" not found Jul 28 01:56:23.002765 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:23.002748 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1b0ca2e-075a-40e7-b328-34d54d43b6d8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-dfed9b-766788bf-md4fq\" (UID: \"c1b0ca2e-075a-40e7-b328-34d54d43b6d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dfed9b-766788bf-md4fq" Jul 28 01:56:23.015198 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:23.015170 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82pjf\" (UniqueName: \"kubernetes.io/projected/c1b0ca2e-075a-40e7-b328-34d54d43b6d8-kube-api-access-82pjf\") pod \"maas-api-e2e-shared-a-dfed9b-766788bf-md4fq\" (UID: \"c1b0ca2e-075a-40e7-b328-34d54d43b6d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dfed9b-766788bf-md4fq" Jul 28 01:56:23.506450 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:23.506409 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1b0ca2e-075a-40e7-b328-34d54d43b6d8-maas-api-tls\") pod \"maas-api-e2e-shared-a-dfed9b-766788bf-md4fq\" (UID: \"c1b0ca2e-075a-40e7-b328-34d54d43b6d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dfed9b-766788bf-md4fq" Jul 28 01:56:23.508767 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:23.508737 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1b0ca2e-075a-40e7-b328-34d54d43b6d8-maas-api-tls\") pod \"maas-api-e2e-shared-a-dfed9b-766788bf-md4fq\" (UID: \"c1b0ca2e-075a-40e7-b328-34d54d43b6d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dfed9b-766788bf-md4fq" Jul 28 01:56:23.704279 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:23.704246 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dfed9b-766788bf-md4fq" Jul 28 01:56:23.833634 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:23.833608 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-dfed9b-766788bf-md4fq"] Jul 28 01:56:23.835818 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:56:23.835786 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b0ca2e_075a_40e7_b328_34d54d43b6d8.slice/crio-6d12212538b2a7c44cc0f572da63f230575db002310016b2ad57eed11c3bbb9c WatchSource:0}: Error finding container 6d12212538b2a7c44cc0f572da63f230575db002310016b2ad57eed11c3bbb9c: Status 404 returned error can't find the container with id 6d12212538b2a7c44cc0f572da63f230575db002310016b2ad57eed11c3bbb9c Jul 28 01:56:23.923095 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:23.923059 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dfed9b-766788bf-md4fq" event={"ID":"c1b0ca2e-075a-40e7-b328-34d54d43b6d8","Type":"ContainerStarted","Data":"6d12212538b2a7c44cc0f572da63f230575db002310016b2ad57eed11c3bbb9c"} Jul 28 01:56:24.930969 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:24.930934 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dfed9b-766788bf-md4fq" event={"ID":"c1b0ca2e-075a-40e7-b328-34d54d43b6d8","Type":"ContainerStarted","Data":"0e4b6cef7f0262a4e21267db86ad1651821f77efa8a62276bc34d6ea72ef2d76"} Jul 28 01:56:24.931426 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:24.931070 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dfed9b-766788bf-md4fq" Jul 28 01:56:24.949972 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:24.949927 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dfed9b-766788bf-md4fq" podStartSLOduration=2.561668461 podStartE2EDuration="2.949912337s" totalImagesPullingTime="388.243876ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:56:24.949059409 +0000 UTC m=+2442.383457721" watchObservedRunningTime="2026-07-28 01:56:24.949912337 +0000 UTC m=+2442.384310652" Jul 28 01:56:27.771056 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:27.771005 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6"] Jul 28 01:56:27.774714 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:27.774695 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:56:27.777900 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:27.777881 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-de398f-openshift-default-dockercfg-fdfp5\"" Jul 28 01:56:27.788858 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:27.788830 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6"] Jul 28 01:56:27.844265 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:27.844226 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f99f783-4ef6-45b4-9692-02b8103b8f53-workload-certs\") pod \"e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6\" (UID: \"8f99f783-4ef6-45b4-9692-02b8103b8f53\") " pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:56:27.844415 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:27.844268 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f99f783-4ef6-45b4-9692-02b8103b8f53-istio-envoy\") pod \"e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6\" (UID: \"8f99f783-4ef6-45b4-9692-02b8103b8f53\") " pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:56:27.844415 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:27.844294 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f99f783-4ef6-45b4-9692-02b8103b8f53-istio-data\") pod \"e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6\" (UID: \"8f99f783-4ef6-45b4-9692-02b8103b8f53\") " pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:56:27.844415 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:27.844364 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq9db\" (UniqueName: \"kubernetes.io/projected/8f99f783-4ef6-45b4-9692-02b8103b8f53-kube-api-access-zq9db\") pod \"e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6\" (UID: \"8f99f783-4ef6-45b4-9692-02b8103b8f53\") " pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:56:27.844415 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:27.844408 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f99f783-4ef6-45b4-9692-02b8103b8f53-credential-socket\") pod \"e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6\" (UID: \"8f99f783-4ef6-45b4-9692-02b8103b8f53\") " pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:56:27.844586 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:27.844463 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f99f783-4ef6-45b4-9692-02b8103b8f53-istio-podinfo\") pod \"e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6\" (UID: \"8f99f783-4ef6-45b4-9692-02b8103b8f53\") " pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:56:27.844586 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:27.844496 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f99f783-4ef6-45b4-9692-02b8103b8f53-workload-socket\") pod \"e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6\" (UID: \"8f99f783-4ef6-45b4-9692-02b8103b8f53\") " pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:56:27.844586 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:27.844515 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f99f783-4ef6-45b4-9692-02b8103b8f53-istio-token\") pod \"e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6\" (UID: \"8f99f783-4ef6-45b4-9692-02b8103b8f53\") " pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:56:27.844586 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:27.844547 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f99f783-4ef6-45b4-9692-02b8103b8f53-istiod-ca-cert\") pod \"e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6\" (UID: \"8f99f783-4ef6-45b4-9692-02b8103b8f53\") " pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:56:27.944982 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:27.944945 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f99f783-4ef6-45b4-9692-02b8103b8f53-istiod-ca-cert\") pod \"e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6\" (UID: \"8f99f783-4ef6-45b4-9692-02b8103b8f53\") " pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:56:27.944982 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:27.944985 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f99f783-4ef6-45b4-9692-02b8103b8f53-workload-certs\") pod \"e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6\" (UID: \"8f99f783-4ef6-45b4-9692-02b8103b8f53\") " pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:56:27.945252 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:27.945001 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f99f783-4ef6-45b4-9692-02b8103b8f53-istio-envoy\") pod \"e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6\" (UID: \"8f99f783-4ef6-45b4-9692-02b8103b8f53\") " pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:56:27.945252 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:27.945016 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f99f783-4ef6-45b4-9692-02b8103b8f53-istio-data\") pod \"e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6\" (UID: \"8f99f783-4ef6-45b4-9692-02b8103b8f53\") " pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:56:27.945252 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:27.945060 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq9db\" (UniqueName: \"kubernetes.io/projected/8f99f783-4ef6-45b4-9692-02b8103b8f53-kube-api-access-zq9db\") pod \"e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6\" (UID: \"8f99f783-4ef6-45b4-9692-02b8103b8f53\") " pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:56:27.945252 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:27.945106 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f99f783-4ef6-45b4-9692-02b8103b8f53-credential-socket\") pod \"e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6\" (UID: \"8f99f783-4ef6-45b4-9692-02b8103b8f53\") " pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:56:27.945252 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:27.945183 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f99f783-4ef6-45b4-9692-02b8103b8f53-istio-podinfo\") pod \"e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6\" (UID: \"8f99f783-4ef6-45b4-9692-02b8103b8f53\") " pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:56:27.945252 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:27.945241 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f99f783-4ef6-45b4-9692-02b8103b8f53-workload-socket\") pod \"e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6\" (UID: \"8f99f783-4ef6-45b4-9692-02b8103b8f53\") " pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:56:27.945560 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:27.945270 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f99f783-4ef6-45b4-9692-02b8103b8f53-istio-token\") pod \"e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6\" (UID: \"8f99f783-4ef6-45b4-9692-02b8103b8f53\") " pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:56:27.945620 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:27.945564 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f99f783-4ef6-45b4-9692-02b8103b8f53-credential-socket\") pod \"e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6\" (UID: \"8f99f783-4ef6-45b4-9692-02b8103b8f53\") " pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:56:27.945620 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:27.945509 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f99f783-4ef6-45b4-9692-02b8103b8f53-istio-data\") pod \"e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6\" (UID: \"8f99f783-4ef6-45b4-9692-02b8103b8f53\") " pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:56:27.945620 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:27.945418 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f99f783-4ef6-45b4-9692-02b8103b8f53-workload-certs\") pod \"e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6\" (UID: \"8f99f783-4ef6-45b4-9692-02b8103b8f53\") " pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:56:27.945777 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:27.945714 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f99f783-4ef6-45b4-9692-02b8103b8f53-workload-socket\") pod \"e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6\" (UID: \"8f99f783-4ef6-45b4-9692-02b8103b8f53\") " pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:56:27.945840 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:27.945778 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f99f783-4ef6-45b4-9692-02b8103b8f53-istiod-ca-cert\") pod \"e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6\" (UID: \"8f99f783-4ef6-45b4-9692-02b8103b8f53\") " pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:56:27.947550 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:27.947530 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f99f783-4ef6-45b4-9692-02b8103b8f53-istio-envoy\") pod \"e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6\" (UID: \"8f99f783-4ef6-45b4-9692-02b8103b8f53\") " pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:56:27.947693 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:27.947675 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f99f783-4ef6-45b4-9692-02b8103b8f53-istio-podinfo\") pod \"e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6\" (UID: \"8f99f783-4ef6-45b4-9692-02b8103b8f53\") " pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:56:27.954635 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:27.954603 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f99f783-4ef6-45b4-9692-02b8103b8f53-istio-token\") pod \"e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6\" (UID: \"8f99f783-4ef6-45b4-9692-02b8103b8f53\") " pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:56:27.954742 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:27.954721 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq9db\" (UniqueName: \"kubernetes.io/projected/8f99f783-4ef6-45b4-9692-02b8103b8f53-kube-api-access-zq9db\") pod \"e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6\" (UID: \"8f99f783-4ef6-45b4-9692-02b8103b8f53\") " pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:56:28.085668 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:28.085589 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:56:28.240426 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:28.240394 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6"] Jul 28 01:56:28.241651 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:56:28.241619 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f99f783_4ef6_45b4_9692_02b8103b8f53.slice/crio-2e76524d7d0e6f8115e161f24f76d6aeeb2c29a66a6a82aaee20203452ec01b3 WatchSource:0}: Error finding container 2e76524d7d0e6f8115e161f24f76d6aeeb2c29a66a6a82aaee20203452ec01b3: Status 404 returned error can't find the container with id 2e76524d7d0e6f8115e161f24f76d6aeeb2c29a66a6a82aaee20203452ec01b3 Jul 28 01:56:28.243808 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:28.243777 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:56:28.243907 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:28.243856 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:56:28.243907 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:28.243900 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:56:28.953095 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:28.953062 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" event={"ID":"8f99f783-4ef6-45b4-9692-02b8103b8f53","Type":"ContainerStarted","Data":"31cae47582fa81a9643776867335507cd32ac50c6e7edf9f7a28e1def90b6908"} Jul 28 01:56:28.953465 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:28.953124 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" event={"ID":"8f99f783-4ef6-45b4-9692-02b8103b8f53","Type":"ContainerStarted","Data":"2e76524d7d0e6f8115e161f24f76d6aeeb2c29a66a6a82aaee20203452ec01b3"} Jul 28 01:56:28.977537 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:28.977488 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" podStartSLOduration=1.977472686 podStartE2EDuration="1.977472686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:56:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:56:28.974803479 +0000 UTC m=+2446.409201814" watchObservedRunningTime="2026-07-28 01:56:28.977472686 +0000 UTC m=+2446.411870999" Jul 28 01:56:29.086540 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:29.086465 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:56:29.092927 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:29.092895 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:56:29.958209 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:29.958180 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:56:29.959068 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:29.959027 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:56:30.941753 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:30.941722 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dfed9b-766788bf-md4fq" Jul 28 01:56:31.109564 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:31.109531 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-de398f-558b47988c-s7nvq"] Jul 28 01:56:31.113093 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:31.113074 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de398f-558b47988c-s7nvq" Jul 28 01:56:31.116130 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:31.116109 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-de398f-serving-cert\"" Jul 28 01:56:31.124282 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:31.124258 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-de398f-558b47988c-s7nvq"] Jul 28 01:56:31.279970 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:31.279877 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6228\" (UniqueName: \"kubernetes.io/projected/95c353c3-777d-4f8e-9eea-3055b77db4d6-kube-api-access-f6228\") pod \"maas-api-e2e-shared-b-de398f-558b47988c-s7nvq\" (UID: \"95c353c3-777d-4f8e-9eea-3055b77db4d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de398f-558b47988c-s7nvq" Jul 28 01:56:31.280145 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:31.280017 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c353c3-777d-4f8e-9eea-3055b77db4d6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-de398f-558b47988c-s7nvq\" (UID: \"95c353c3-777d-4f8e-9eea-3055b77db4d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de398f-558b47988c-s7nvq" Jul 28 01:56:31.280145 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:31.280082 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95c353c3-777d-4f8e-9eea-3055b77db4d6-maas-api-tls\") pod \"maas-api-e2e-shared-b-de398f-558b47988c-s7nvq\" (UID: \"95c353c3-777d-4f8e-9eea-3055b77db4d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de398f-558b47988c-s7nvq" Jul 28 01:56:31.381517 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:31.381483 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6228\" (UniqueName: \"kubernetes.io/projected/95c353c3-777d-4f8e-9eea-3055b77db4d6-kube-api-access-f6228\") pod \"maas-api-e2e-shared-b-de398f-558b47988c-s7nvq\" (UID: \"95c353c3-777d-4f8e-9eea-3055b77db4d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de398f-558b47988c-s7nvq" Jul 28 01:56:31.381681 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:31.381545 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c353c3-777d-4f8e-9eea-3055b77db4d6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-de398f-558b47988c-s7nvq\" (UID: \"95c353c3-777d-4f8e-9eea-3055b77db4d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de398f-558b47988c-s7nvq" Jul 28 01:56:31.381681 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:31.381575 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95c353c3-777d-4f8e-9eea-3055b77db4d6-maas-api-tls\") pod \"maas-api-e2e-shared-b-de398f-558b47988c-s7nvq\" (UID: \"95c353c3-777d-4f8e-9eea-3055b77db4d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de398f-558b47988c-s7nvq" Jul 28 01:56:31.381761 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:56:31.381679 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-de398f-serving-cert: secret "maas-api-e2e-shared-b-de398f-serving-cert" not found Jul 28 01:56:31.381761 ip-10-0-131-23 kubenswrapper[2565]: E0728 01:56:31.381751 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95c353c3-777d-4f8e-9eea-3055b77db4d6-maas-api-tls podName:95c353c3-777d-4f8e-9eea-3055b77db4d6 nodeName:}" failed. No retries permitted until 2026-07-28 01:56:31.881732004 +0000 UTC m=+2449.316130326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/95c353c3-777d-4f8e-9eea-3055b77db4d6-maas-api-tls") pod "maas-api-e2e-shared-b-de398f-558b47988c-s7nvq" (UID: "95c353c3-777d-4f8e-9eea-3055b77db4d6") : secret "maas-api-e2e-shared-b-de398f-serving-cert" not found Jul 28 01:56:31.382173 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:31.382153 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c353c3-777d-4f8e-9eea-3055b77db4d6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-de398f-558b47988c-s7nvq\" (UID: \"95c353c3-777d-4f8e-9eea-3055b77db4d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de398f-558b47988c-s7nvq" Jul 28 01:56:31.396058 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:31.396006 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6228\" (UniqueName: \"kubernetes.io/projected/95c353c3-777d-4f8e-9eea-3055b77db4d6-kube-api-access-f6228\") pod \"maas-api-e2e-shared-b-de398f-558b47988c-s7nvq\" (UID: \"95c353c3-777d-4f8e-9eea-3055b77db4d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de398f-558b47988c-s7nvq" Jul 28 01:56:31.885610 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:31.885576 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95c353c3-777d-4f8e-9eea-3055b77db4d6-maas-api-tls\") pod \"maas-api-e2e-shared-b-de398f-558b47988c-s7nvq\" (UID: \"95c353c3-777d-4f8e-9eea-3055b77db4d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de398f-558b47988c-s7nvq" Jul 28 01:56:31.887939 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:31.887909 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95c353c3-777d-4f8e-9eea-3055b77db4d6-maas-api-tls\") pod \"maas-api-e2e-shared-b-de398f-558b47988c-s7nvq\" (UID: \"95c353c3-777d-4f8e-9eea-3055b77db4d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de398f-558b47988c-s7nvq" Jul 28 01:56:32.023882 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:32.023857 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de398f-558b47988c-s7nvq" Jul 28 01:56:32.165646 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:32.165573 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-de398f-558b47988c-s7nvq"] Jul 28 01:56:32.172606 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:56:32.172577 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95c353c3_777d_4f8e_9eea_3055b77db4d6.slice/crio-9682ea5c5379b7701dabe5195fbdf194c0a365f88acb6326332a882cc4284efc WatchSource:0}: Error finding container 9682ea5c5379b7701dabe5195fbdf194c0a365f88acb6326332a882cc4284efc: Status 404 returned error can't find the container with id 9682ea5c5379b7701dabe5195fbdf194c0a365f88acb6326332a882cc4284efc Jul 28 01:56:32.974055 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:32.973987 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de398f-558b47988c-s7nvq" event={"ID":"95c353c3-777d-4f8e-9eea-3055b77db4d6","Type":"ContainerStarted","Data":"eb5e1da5aa565ef23ba23d3ed4392e79e5bb6c87ca3f4f8f272568444aad53d1"} Jul 28 01:56:32.974055 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:32.974054 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de398f-558b47988c-s7nvq" event={"ID":"95c353c3-777d-4f8e-9eea-3055b77db4d6","Type":"ContainerStarted","Data":"9682ea5c5379b7701dabe5195fbdf194c0a365f88acb6326332a882cc4284efc"} Jul 28 01:56:32.974282 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:32.974086 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de398f-558b47988c-s7nvq" Jul 28 01:56:32.999663 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:32.999618 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de398f-558b47988c-s7nvq" podStartSLOduration=1.684478812 podStartE2EDuration="1.999604037s" totalImagesPullingTime="315.125225ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:56:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:56:32.999165305 +0000 UTC m=+2450.433563618" watchObservedRunningTime="2026-07-28 01:56:32.999604037 +0000 UTC m=+2450.434002350" Jul 28 01:56:38.984052 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:38.983996 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de398f-558b47988c-s7nvq" Jul 28 01:56:43.034405 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:43.034322 2565 generic.go:356] "Generic (PLEG): container finished" podID="8f99f783-4ef6-45b4-9692-02b8103b8f53" containerID="31cae47582fa81a9643776867335507cd32ac50c6e7edf9f7a28e1def90b6908" exitCode=0 Jul 28 01:56:43.034405 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:43.034377 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" event={"ID":"8f99f783-4ef6-45b4-9692-02b8103b8f53","Type":"ContainerDied","Data":"31cae47582fa81a9643776867335507cd32ac50c6e7edf9f7a28e1def90b6908"} Jul 28 01:56:43.034832 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:43.034724 2565 scope.go:117] "RemoveContainer" containerID="31cae47582fa81a9643776867335507cd32ac50c6e7edf9f7a28e1def90b6908" Jul 28 01:56:43.035765 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:43.035732 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:56:43.035838 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:43.035820 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:56:43.035884 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:43.035870 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:56:44.042363 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:44.042327 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" event={"ID":"8f99f783-4ef6-45b4-9692-02b8103b8f53","Type":"ContainerStarted","Data":"efc1163baa24546df5efbbc1183c548232bfbabdd27f2e366284bba76938f34e"} Jul 28 01:56:44.086560 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:44.086523 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:56:44.087487 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:44.087460 2565 patch_prober.go:28] interesting pod/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 28 01:56:44.087635 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:44.087520 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" podUID="8f99f783-4ef6-45b4-9692-02b8103b8f53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 28 01:56:44.958992 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:44.958957 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:56:45.086779 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:45.086748 2565 patch_prober.go:28] interesting pod/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 28 01:56:45.087171 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:45.086799 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" podUID="8f99f783-4ef6-45b4-9692-02b8103b8f53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 28 01:56:46.086608 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:46.086571 2565 patch_prober.go:28] interesting pod/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 28 01:56:46.086795 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:46.086625 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" podUID="8f99f783-4ef6-45b4-9692-02b8103b8f53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 28 01:56:47.086270 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:47.086233 2565 patch_prober.go:28] interesting pod/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 28 01:56:47.086462 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:47.086291 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" podUID="8f99f783-4ef6-45b4-9692-02b8103b8f53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 28 01:56:48.086880 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:48.086848 2565 patch_prober.go:28] interesting pod/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 28 01:56:48.087324 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:48.086902 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" podUID="8f99f783-4ef6-45b4-9692-02b8103b8f53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 28 01:56:49.090157 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:49.090126 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:56:49.091069 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:56:49.091048 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 01:57:08.713441 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:57:08.713408 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5"] Jul 28 01:57:08.716960 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:57:08.716938 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5" Jul 28 01:57:08.721730 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:57:08.721704 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-de398f\"/\"default-dockercfg-wrrtm\"" Jul 28 01:57:08.721877 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:57:08.721755 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-de398f\"/\"auth-test-model-de398f-kserve-self-signed-certs\"" Jul 28 01:57:08.721877 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:57:08.721760 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-de398f\"/\"openshift-service-ca.crt\"" Jul 28 01:57:08.721877 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:57:08.721831 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-de398f\"/\"kube-root-ca.crt\"" Jul 28 01:57:08.733660 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:57:08.733628 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5"] Jul 28 01:57:08.823112 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:57:08.823071 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e497050-ccb6-4cf3-a4ad-e00f678c6d73-model-cache\") pod \"auth-test-model-de398f-kserve-564c87bf6c-gknf5\" (UID: \"7e497050-ccb6-4cf3-a4ad-e00f678c6d73\") " pod="ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5" Jul 28 01:57:08.823112 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:57:08.823111 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e497050-ccb6-4cf3-a4ad-e00f678c6d73-tmp-dir\") pod \"auth-test-model-de398f-kserve-564c87bf6c-gknf5\" (UID: \"7e497050-ccb6-4cf3-a4ad-e00f678c6d73\") " pod="ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5" Jul 28 01:57:08.823363 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:57:08.823198 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e497050-ccb6-4cf3-a4ad-e00f678c6d73-dshm\") pod \"auth-test-model-de398f-kserve-564c87bf6c-gknf5\" (UID: \"7e497050-ccb6-4cf3-a4ad-e00f678c6d73\") " pod="ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5" Jul 28 01:57:08.823363 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:57:08.823229 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e497050-ccb6-4cf3-a4ad-e00f678c6d73-tls-certs\") pod \"auth-test-model-de398f-kserve-564c87bf6c-gknf5\" (UID: \"7e497050-ccb6-4cf3-a4ad-e00f678c6d73\") " pod="ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5" Jul 28 01:57:08.823363 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:57:08.823246 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e497050-ccb6-4cf3-a4ad-e00f678c6d73-home\") pod \"auth-test-model-de398f-kserve-564c87bf6c-gknf5\" (UID: \"7e497050-ccb6-4cf3-a4ad-e00f678c6d73\") " pod="ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5" Jul 28 01:57:08.823514 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:57:08.823387 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vkqn\" (UniqueName: \"kubernetes.io/projected/7e497050-ccb6-4cf3-a4ad-e00f678c6d73-kube-api-access-8vkqn\") pod \"auth-test-model-de398f-kserve-564c87bf6c-gknf5\" (UID: \"7e497050-ccb6-4cf3-a4ad-e00f678c6d73\") " pod="ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5" Jul 28 01:57:08.924401 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:57:08.924364 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e497050-ccb6-4cf3-a4ad-e00f678c6d73-model-cache\") pod \"auth-test-model-de398f-kserve-564c87bf6c-gknf5\" (UID: \"7e497050-ccb6-4cf3-a4ad-e00f678c6d73\") " pod="ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5" Jul 28 01:57:08.924401 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:57:08.924402 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e497050-ccb6-4cf3-a4ad-e00f678c6d73-tmp-dir\") pod \"auth-test-model-de398f-kserve-564c87bf6c-gknf5\" (UID: \"7e497050-ccb6-4cf3-a4ad-e00f678c6d73\") " pod="ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5" Jul 28 01:57:08.924654 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:57:08.924459 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e497050-ccb6-4cf3-a4ad-e00f678c6d73-dshm\") pod \"auth-test-model-de398f-kserve-564c87bf6c-gknf5\" (UID: \"7e497050-ccb6-4cf3-a4ad-e00f678c6d73\") " pod="ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5" Jul 28 01:57:08.924654 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:57:08.924481 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e497050-ccb6-4cf3-a4ad-e00f678c6d73-tls-certs\") pod \"auth-test-model-de398f-kserve-564c87bf6c-gknf5\" (UID: \"7e497050-ccb6-4cf3-a4ad-e00f678c6d73\") " pod="ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5" Jul 28 01:57:08.924654 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:57:08.924504 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e497050-ccb6-4cf3-a4ad-e00f678c6d73-home\") pod \"auth-test-model-de398f-kserve-564c87bf6c-gknf5\" (UID: \"7e497050-ccb6-4cf3-a4ad-e00f678c6d73\") " pod="ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5" Jul 28 01:57:08.924654 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:57:08.924591 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vkqn\" (UniqueName: \"kubernetes.io/projected/7e497050-ccb6-4cf3-a4ad-e00f678c6d73-kube-api-access-8vkqn\") pod \"auth-test-model-de398f-kserve-564c87bf6c-gknf5\" (UID: \"7e497050-ccb6-4cf3-a4ad-e00f678c6d73\") " pod="ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5" Jul 28 01:57:08.924857 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:57:08.924825 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e497050-ccb6-4cf3-a4ad-e00f678c6d73-model-cache\") pod \"auth-test-model-de398f-kserve-564c87bf6c-gknf5\" (UID: \"7e497050-ccb6-4cf3-a4ad-e00f678c6d73\") " pod="ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5" Jul 28 01:57:08.924914 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:57:08.924828 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e497050-ccb6-4cf3-a4ad-e00f678c6d73-tmp-dir\") pod \"auth-test-model-de398f-kserve-564c87bf6c-gknf5\" (UID: \"7e497050-ccb6-4cf3-a4ad-e00f678c6d73\") " pod="ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5" Jul 28 01:57:08.924970 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:57:08.924909 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e497050-ccb6-4cf3-a4ad-e00f678c6d73-home\") pod \"auth-test-model-de398f-kserve-564c87bf6c-gknf5\" (UID: \"7e497050-ccb6-4cf3-a4ad-e00f678c6d73\") " pod="ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5" Jul 28 01:57:08.926793 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:57:08.926760 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e497050-ccb6-4cf3-a4ad-e00f678c6d73-dshm\") pod \"auth-test-model-de398f-kserve-564c87bf6c-gknf5\" (UID: \"7e497050-ccb6-4cf3-a4ad-e00f678c6d73\") " pod="ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5" Jul 28 01:57:08.926982 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:57:08.926963 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e497050-ccb6-4cf3-a4ad-e00f678c6d73-tls-certs\") pod \"auth-test-model-de398f-kserve-564c87bf6c-gknf5\" (UID: \"7e497050-ccb6-4cf3-a4ad-e00f678c6d73\") " pod="ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5" Jul 28 01:57:08.936199 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:57:08.936171 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vkqn\" (UniqueName: \"kubernetes.io/projected/7e497050-ccb6-4cf3-a4ad-e00f678c6d73-kube-api-access-8vkqn\") pod \"auth-test-model-de398f-kserve-564c87bf6c-gknf5\" (UID: \"7e497050-ccb6-4cf3-a4ad-e00f678c6d73\") " pod="ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5" Jul 28 01:57:09.025901 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:57:09.025828 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5" Jul 28 01:57:09.164844 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:57:09.164804 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5"] Jul 28 01:57:09.169535 ip-10-0-131-23 kubenswrapper[2565]: W0728 01:57:09.169505 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e497050_ccb6_4cf3_a4ad_e00f678c6d73.slice/crio-1007f42de859e765d21b334644669124cacdbd15ee5e4cb11140d9e11f3c5d02 WatchSource:0}: Error finding container 1007f42de859e765d21b334644669124cacdbd15ee5e4cb11140d9e11f3c5d02: Status 404 returned error can't find the container with id 1007f42de859e765d21b334644669124cacdbd15ee5e4cb11140d9e11f3c5d02 Jul 28 01:57:09.176211 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:57:09.176186 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5" event={"ID":"7e497050-ccb6-4cf3-a4ad-e00f678c6d73","Type":"ContainerStarted","Data":"1007f42de859e765d21b334644669124cacdbd15ee5e4cb11140d9e11f3c5d02"} Jul 28 01:57:10.182755 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:57:10.182716 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5" event={"ID":"7e497050-ccb6-4cf3-a4ad-e00f678c6d73","Type":"ContainerStarted","Data":"cde3ece8a6560dd6d5a7c33ed9b61056b541600758a0299e29cdcbb7d4d2ad02"} Jul 28 01:57:10.207273 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:57:10.207226 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5" podStartSLOduration=2.207211049 podStartE2EDuration="2.207211049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:57:10.205911815 +0000 UTC m=+2487.640310128" watchObservedRunningTime="2026-07-28 01:57:10.207211049 +0000 UTC m=+2487.641609362" Jul 28 01:57:19.026743 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:57:19.026707 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5" Jul 28 01:57:19.027331 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:57:19.026756 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5" Jul 28 01:57:19.038964 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:57:19.038945 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5" Jul 28 01:57:19.238939 ip-10-0-131-23 kubenswrapper[2565]: I0728 01:57:19.238913 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5" Jul 28 02:01:19.068490 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:01:19.068456 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5"] Jul 28 02:01:34.086818 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:01:34.086758 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5" podUID="7e497050-ccb6-4cf3-a4ad-e00f678c6d73" containerName="main" containerID="cri-o://cde3ece8a6560dd6d5a7c33ed9b61056b541600758a0299e29cdcbb7d4d2ad02" gracePeriod=45 Jul 28 02:01:34.327581 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:01:34.327546 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5" Jul 28 02:01:34.414186 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:01:34.414088 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vkqn\" (UniqueName: \"kubernetes.io/projected/7e497050-ccb6-4cf3-a4ad-e00f678c6d73-kube-api-access-8vkqn\") pod \"7e497050-ccb6-4cf3-a4ad-e00f678c6d73\" (UID: \"7e497050-ccb6-4cf3-a4ad-e00f678c6d73\") " Jul 28 02:01:34.414186 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:01:34.414145 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e497050-ccb6-4cf3-a4ad-e00f678c6d73-home\") pod \"7e497050-ccb6-4cf3-a4ad-e00f678c6d73\" (UID: \"7e497050-ccb6-4cf3-a4ad-e00f678c6d73\") " Jul 28 02:01:34.414484 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:01:34.414211 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e497050-ccb6-4cf3-a4ad-e00f678c6d73-model-cache\") pod \"7e497050-ccb6-4cf3-a4ad-e00f678c6d73\" (UID: \"7e497050-ccb6-4cf3-a4ad-e00f678c6d73\") " Jul 28 02:01:34.414484 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:01:34.414252 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e497050-ccb6-4cf3-a4ad-e00f678c6d73-dshm\") pod \"7e497050-ccb6-4cf3-a4ad-e00f678c6d73\" (UID: \"7e497050-ccb6-4cf3-a4ad-e00f678c6d73\") " Jul 28 02:01:34.414484 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:01:34.414276 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e497050-ccb6-4cf3-a4ad-e00f678c6d73-tls-certs\") pod \"7e497050-ccb6-4cf3-a4ad-e00f678c6d73\" (UID: \"7e497050-ccb6-4cf3-a4ad-e00f678c6d73\") " Jul 28 02:01:34.414484 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:01:34.414350 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e497050-ccb6-4cf3-a4ad-e00f678c6d73-tmp-dir\") pod \"7e497050-ccb6-4cf3-a4ad-e00f678c6d73\" (UID: \"7e497050-ccb6-4cf3-a4ad-e00f678c6d73\") " Jul 28 02:01:34.414683 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:01:34.414471 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e497050-ccb6-4cf3-a4ad-e00f678c6d73-home" (OuterVolumeSpecName: "home") pod "7e497050-ccb6-4cf3-a4ad-e00f678c6d73" (UID: "7e497050-ccb6-4cf3-a4ad-e00f678c6d73"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 02:01:34.414683 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:01:34.414488 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e497050-ccb6-4cf3-a4ad-e00f678c6d73-model-cache" (OuterVolumeSpecName: "model-cache") pod "7e497050-ccb6-4cf3-a4ad-e00f678c6d73" (UID: "7e497050-ccb6-4cf3-a4ad-e00f678c6d73"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 02:01:34.414789 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:01:34.414679 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e497050-ccb6-4cf3-a4ad-e00f678c6d73-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7e497050-ccb6-4cf3-a4ad-e00f678c6d73" (UID: "7e497050-ccb6-4cf3-a4ad-e00f678c6d73"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 02:01:34.414789 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:01:34.414756 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e497050-ccb6-4cf3-a4ad-e00f678c6d73-home\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:01:34.414789 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:01:34.414775 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e497050-ccb6-4cf3-a4ad-e00f678c6d73-model-cache\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:01:34.416419 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:01:34.416395 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e497050-ccb6-4cf3-a4ad-e00f678c6d73-dshm" (OuterVolumeSpecName: "dshm") pod "7e497050-ccb6-4cf3-a4ad-e00f678c6d73" (UID: "7e497050-ccb6-4cf3-a4ad-e00f678c6d73"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 02:01:34.416517 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:01:34.416433 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e497050-ccb6-4cf3-a4ad-e00f678c6d73-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e497050-ccb6-4cf3-a4ad-e00f678c6d73" (UID: "7e497050-ccb6-4cf3-a4ad-e00f678c6d73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 02:01:34.416517 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:01:34.416464 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e497050-ccb6-4cf3-a4ad-e00f678c6d73-kube-api-access-8vkqn" (OuterVolumeSpecName: "kube-api-access-8vkqn") pod "7e497050-ccb6-4cf3-a4ad-e00f678c6d73" (UID: "7e497050-ccb6-4cf3-a4ad-e00f678c6d73"). InnerVolumeSpecName "kube-api-access-8vkqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 02:01:34.515815 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:01:34.515786 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e497050-ccb6-4cf3-a4ad-e00f678c6d73-dshm\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:01:34.515815 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:01:34.515811 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e497050-ccb6-4cf3-a4ad-e00f678c6d73-tls-certs\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:01:34.515815 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:01:34.515822 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e497050-ccb6-4cf3-a4ad-e00f678c6d73-tmp-dir\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:01:34.516065 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:01:34.515831 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vkqn\" (UniqueName: \"kubernetes.io/projected/7e497050-ccb6-4cf3-a4ad-e00f678c6d73-kube-api-access-8vkqn\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:01:34.550654 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:01:34.550614 2565 generic.go:356] "Generic (PLEG): container finished" podID="7e497050-ccb6-4cf3-a4ad-e00f678c6d73" containerID="cde3ece8a6560dd6d5a7c33ed9b61056b541600758a0299e29cdcbb7d4d2ad02" exitCode=0 Jul 28 02:01:34.550827 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:01:34.550659 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5" Jul 28 02:01:34.550827 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:01:34.550666 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5" event={"ID":"7e497050-ccb6-4cf3-a4ad-e00f678c6d73","Type":"ContainerDied","Data":"cde3ece8a6560dd6d5a7c33ed9b61056b541600758a0299e29cdcbb7d4d2ad02"} Jul 28 02:01:34.550827 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:01:34.550699 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5" event={"ID":"7e497050-ccb6-4cf3-a4ad-e00f678c6d73","Type":"ContainerDied","Data":"1007f42de859e765d21b334644669124cacdbd15ee5e4cb11140d9e11f3c5d02"} Jul 28 02:01:34.550827 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:01:34.550717 2565 scope.go:117] "RemoveContainer" containerID="cde3ece8a6560dd6d5a7c33ed9b61056b541600758a0299e29cdcbb7d4d2ad02" Jul 28 02:01:34.562003 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:01:34.561964 2565 scope.go:117] "RemoveContainer" containerID="cde3ece8a6560dd6d5a7c33ed9b61056b541600758a0299e29cdcbb7d4d2ad02" Jul 28 02:01:34.562379 ip-10-0-131-23 kubenswrapper[2565]: E0728 02:01:34.562356 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cde3ece8a6560dd6d5a7c33ed9b61056b541600758a0299e29cdcbb7d4d2ad02\": container with ID starting with cde3ece8a6560dd6d5a7c33ed9b61056b541600758a0299e29cdcbb7d4d2ad02 not found: ID does not exist" containerID="cde3ece8a6560dd6d5a7c33ed9b61056b541600758a0299e29cdcbb7d4d2ad02" Jul 28 02:01:34.562449 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:01:34.562394 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cde3ece8a6560dd6d5a7c33ed9b61056b541600758a0299e29cdcbb7d4d2ad02"} err="failed to get container status \"cde3ece8a6560dd6d5a7c33ed9b61056b541600758a0299e29cdcbb7d4d2ad02\": rpc error: code = NotFound desc = could not find container \"cde3ece8a6560dd6d5a7c33ed9b61056b541600758a0299e29cdcbb7d4d2ad02\": container with ID starting with cde3ece8a6560dd6d5a7c33ed9b61056b541600758a0299e29cdcbb7d4d2ad02 not found: ID does not exist" Jul 28 02:01:34.572616 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:01:34.572588 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5"] Jul 28 02:01:34.576462 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:01:34.576434 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-de398f/auth-test-model-de398f-kserve-564c87bf6c-gknf5"] Jul 28 02:01:35.101309 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:01:35.101271 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e497050-ccb6-4cf3-a4ad-e00f678c6d73" path="/var/lib/kubelet/pods/7e497050-ccb6-4cf3-a4ad-e00f678c6d73/volumes" Jul 28 02:01:43.360097 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:01:43.360065 2565 scope.go:117] "RemoveContainer" containerID="ed0f66498da0d6b78d574fab1d18e395182fac0c18ddf398646e8baeab6899cf" Jul 28 02:02:17.519946 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:17.519874 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q"] Jul 28 02:02:17.520405 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:17.520240 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e497050-ccb6-4cf3-a4ad-e00f678c6d73" containerName="main" Jul 28 02:02:17.520405 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:17.520252 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e497050-ccb6-4cf3-a4ad-e00f678c6d73" containerName="main" Jul 28 02:02:17.520405 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:17.520321 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e497050-ccb6-4cf3-a4ad-e00f678c6d73" containerName="main" Jul 28 02:02:17.523403 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:17.523388 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q" Jul 28 02:02:17.527502 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:17.527478 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-de398f\"/\"openshift-service-ca.crt\"" Jul 28 02:02:17.527502 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:17.527478 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-de398f\"/\"default-dockercfg-wrrtm\"" Jul 28 02:02:17.527712 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:17.527534 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-de398f\"/\"sub-test-model-de398f-kserve-self-signed-certs\"" Jul 28 02:02:17.527712 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:17.527607 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-de398f\"/\"kube-root-ca.crt\"" Jul 28 02:02:17.537833 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:17.537814 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q"] Jul 28 02:02:17.598012 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:17.597969 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68bc7ebe-9acd-46f6-8424-5dacbada4945-dshm\") pod \"sub-test-model-de398f-kserve-5fb758cdbf-cnv5q\" (UID: \"68bc7ebe-9acd-46f6-8424-5dacbada4945\") " pod="ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q" Jul 28 02:02:17.598204 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:17.598017 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgdpg\" (UniqueName: \"kubernetes.io/projected/68bc7ebe-9acd-46f6-8424-5dacbada4945-kube-api-access-pgdpg\") pod \"sub-test-model-de398f-kserve-5fb758cdbf-cnv5q\" (UID: \"68bc7ebe-9acd-46f6-8424-5dacbada4945\") " pod="ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q" Jul 28 02:02:17.598204 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:17.598111 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68bc7ebe-9acd-46f6-8424-5dacbada4945-home\") pod \"sub-test-model-de398f-kserve-5fb758cdbf-cnv5q\" (UID: \"68bc7ebe-9acd-46f6-8424-5dacbada4945\") " pod="ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q" Jul 28 02:02:17.598204 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:17.598166 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68bc7ebe-9acd-46f6-8424-5dacbada4945-tmp-dir\") pod \"sub-test-model-de398f-kserve-5fb758cdbf-cnv5q\" (UID: \"68bc7ebe-9acd-46f6-8424-5dacbada4945\") " pod="ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q" Jul 28 02:02:17.598204 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:17.598201 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68bc7ebe-9acd-46f6-8424-5dacbada4945-model-cache\") pod \"sub-test-model-de398f-kserve-5fb758cdbf-cnv5q\" (UID: \"68bc7ebe-9acd-46f6-8424-5dacbada4945\") " pod="ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q" Jul 28 02:02:17.598375 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:17.598272 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68bc7ebe-9acd-46f6-8424-5dacbada4945-tls-certs\") pod \"sub-test-model-de398f-kserve-5fb758cdbf-cnv5q\" (UID: \"68bc7ebe-9acd-46f6-8424-5dacbada4945\") " pod="ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q" Jul 28 02:02:17.699283 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:17.699250 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68bc7ebe-9acd-46f6-8424-5dacbada4945-tls-certs\") pod \"sub-test-model-de398f-kserve-5fb758cdbf-cnv5q\" (UID: \"68bc7ebe-9acd-46f6-8424-5dacbada4945\") " pod="ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q" Jul 28 02:02:17.699447 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:17.699307 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68bc7ebe-9acd-46f6-8424-5dacbada4945-dshm\") pod \"sub-test-model-de398f-kserve-5fb758cdbf-cnv5q\" (UID: \"68bc7ebe-9acd-46f6-8424-5dacbada4945\") " pod="ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q" Jul 28 02:02:17.699447 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:17.699332 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgdpg\" (UniqueName: \"kubernetes.io/projected/68bc7ebe-9acd-46f6-8424-5dacbada4945-kube-api-access-pgdpg\") pod \"sub-test-model-de398f-kserve-5fb758cdbf-cnv5q\" (UID: \"68bc7ebe-9acd-46f6-8424-5dacbada4945\") " pod="ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q" Jul 28 02:02:17.699447 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:17.699362 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68bc7ebe-9acd-46f6-8424-5dacbada4945-home\") pod \"sub-test-model-de398f-kserve-5fb758cdbf-cnv5q\" (UID: \"68bc7ebe-9acd-46f6-8424-5dacbada4945\") " pod="ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q" Jul 28 02:02:17.699447 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:17.699414 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68bc7ebe-9acd-46f6-8424-5dacbada4945-tmp-dir\") pod \"sub-test-model-de398f-kserve-5fb758cdbf-cnv5q\" (UID: \"68bc7ebe-9acd-46f6-8424-5dacbada4945\") " pod="ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q" Jul 28 02:02:17.699626 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:17.699457 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68bc7ebe-9acd-46f6-8424-5dacbada4945-model-cache\") pod \"sub-test-model-de398f-kserve-5fb758cdbf-cnv5q\" (UID: \"68bc7ebe-9acd-46f6-8424-5dacbada4945\") " pod="ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q" Jul 28 02:02:17.699813 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:17.699791 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68bc7ebe-9acd-46f6-8424-5dacbada4945-home\") pod \"sub-test-model-de398f-kserve-5fb758cdbf-cnv5q\" (UID: \"68bc7ebe-9acd-46f6-8424-5dacbada4945\") " pod="ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q" Jul 28 02:02:17.699904 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:17.699838 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68bc7ebe-9acd-46f6-8424-5dacbada4945-model-cache\") pod \"sub-test-model-de398f-kserve-5fb758cdbf-cnv5q\" (UID: \"68bc7ebe-9acd-46f6-8424-5dacbada4945\") " pod="ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q" Jul 28 02:02:17.699904 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:17.699885 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68bc7ebe-9acd-46f6-8424-5dacbada4945-tmp-dir\") pod \"sub-test-model-de398f-kserve-5fb758cdbf-cnv5q\" (UID: \"68bc7ebe-9acd-46f6-8424-5dacbada4945\") " pod="ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q" Jul 28 02:02:17.701657 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:17.701638 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68bc7ebe-9acd-46f6-8424-5dacbada4945-dshm\") pod \"sub-test-model-de398f-kserve-5fb758cdbf-cnv5q\" (UID: \"68bc7ebe-9acd-46f6-8424-5dacbada4945\") " pod="ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q" Jul 28 02:02:17.701911 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:17.701894 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68bc7ebe-9acd-46f6-8424-5dacbada4945-tls-certs\") pod \"sub-test-model-de398f-kserve-5fb758cdbf-cnv5q\" (UID: \"68bc7ebe-9acd-46f6-8424-5dacbada4945\") " pod="ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q" Jul 28 02:02:17.709015 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:17.708987 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgdpg\" (UniqueName: \"kubernetes.io/projected/68bc7ebe-9acd-46f6-8424-5dacbada4945-kube-api-access-pgdpg\") pod \"sub-test-model-de398f-kserve-5fb758cdbf-cnv5q\" (UID: \"68bc7ebe-9acd-46f6-8424-5dacbada4945\") " pod="ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q" Jul 28 02:02:17.832020 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:17.831928 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q" Jul 28 02:02:17.966590 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:17.966567 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q"] Jul 28 02:02:17.968441 ip-10-0-131-23 kubenswrapper[2565]: W0728 02:02:17.968405 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68bc7ebe_9acd_46f6_8424_5dacbada4945.slice/crio-7f4d0563450752bcf4013d07362066038380feafc3e39dbd0e943f1b2be65a88 WatchSource:0}: Error finding container 7f4d0563450752bcf4013d07362066038380feafc3e39dbd0e943f1b2be65a88: Status 404 returned error can't find the container with id 7f4d0563450752bcf4013d07362066038380feafc3e39dbd0e943f1b2be65a88 Jul 28 02:02:18.771619 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:18.771581 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q" event={"ID":"68bc7ebe-9acd-46f6-8424-5dacbada4945","Type":"ContainerStarted","Data":"c75765aeefa9aaf44cdcd0242117c04ab0628417e338b3862a7139c3e3fae04b"} Jul 28 02:02:18.771619 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:18.771624 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q" event={"ID":"68bc7ebe-9acd-46f6-8424-5dacbada4945","Type":"ContainerStarted","Data":"7f4d0563450752bcf4013d07362066038380feafc3e39dbd0e943f1b2be65a88"} Jul 28 02:02:18.794098 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:18.794047 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q" podStartSLOduration=1.794021019 podStartE2EDuration="1.794021019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:02:18.791352652 +0000 UTC m=+2796.225750967" watchObservedRunningTime="2026-07-28 02:02:18.794021019 +0000 UTC m=+2796.228419332" Jul 28 02:02:27.832661 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:27.832630 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q" Jul 28 02:02:27.833163 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:27.832704 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q" Jul 28 02:02:27.845488 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:27.845469 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q" Jul 28 02:02:28.832754 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:28.832724 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q" Jul 28 02:02:43.383154 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:02:43.383121 2565 scope.go:117] "RemoveContainer" containerID="94ec1b91e6a07224e685efe61f39eac510b413d690991f9edfb1b1790351b51d" Jul 28 02:03:19.553395 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:03:19.553364 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q"] Jul 28 02:03:34.571695 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:03:34.571647 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q" podUID="68bc7ebe-9acd-46f6-8424-5dacbada4945" containerName="main" containerID="cri-o://c75765aeefa9aaf44cdcd0242117c04ab0628417e338b3862a7139c3e3fae04b" gracePeriod=45 Jul 28 02:03:34.809388 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:03:34.809363 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q" Jul 28 02:03:34.981392 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:03:34.981350 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68bc7ebe-9acd-46f6-8424-5dacbada4945-dshm\") pod \"68bc7ebe-9acd-46f6-8424-5dacbada4945\" (UID: \"68bc7ebe-9acd-46f6-8424-5dacbada4945\") " Jul 28 02:03:34.981617 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:03:34.981456 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgdpg\" (UniqueName: \"kubernetes.io/projected/68bc7ebe-9acd-46f6-8424-5dacbada4945-kube-api-access-pgdpg\") pod \"68bc7ebe-9acd-46f6-8424-5dacbada4945\" (UID: \"68bc7ebe-9acd-46f6-8424-5dacbada4945\") " Jul 28 02:03:34.981617 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:03:34.981483 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68bc7ebe-9acd-46f6-8424-5dacbada4945-model-cache\") pod \"68bc7ebe-9acd-46f6-8424-5dacbada4945\" (UID: \"68bc7ebe-9acd-46f6-8424-5dacbada4945\") " Jul 28 02:03:34.981725 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:03:34.981643 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68bc7ebe-9acd-46f6-8424-5dacbada4945-tls-certs\") pod \"68bc7ebe-9acd-46f6-8424-5dacbada4945\" (UID: \"68bc7ebe-9acd-46f6-8424-5dacbada4945\") " Jul 28 02:03:34.981725 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:03:34.981696 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68bc7ebe-9acd-46f6-8424-5dacbada4945-home\") pod \"68bc7ebe-9acd-46f6-8424-5dacbada4945\" (UID: \"68bc7ebe-9acd-46f6-8424-5dacbada4945\") " Jul 28 02:03:34.981824 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:03:34.981736 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68bc7ebe-9acd-46f6-8424-5dacbada4945-tmp-dir\") pod \"68bc7ebe-9acd-46f6-8424-5dacbada4945\" (UID: \"68bc7ebe-9acd-46f6-8424-5dacbada4945\") " Jul 28 02:03:34.981824 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:03:34.981744 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bc7ebe-9acd-46f6-8424-5dacbada4945-model-cache" (OuterVolumeSpecName: "model-cache") pod "68bc7ebe-9acd-46f6-8424-5dacbada4945" (UID: "68bc7ebe-9acd-46f6-8424-5dacbada4945"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 02:03:34.982021 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:03:34.982000 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bc7ebe-9acd-46f6-8424-5dacbada4945-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "68bc7ebe-9acd-46f6-8424-5dacbada4945" (UID: "68bc7ebe-9acd-46f6-8424-5dacbada4945"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 02:03:34.982113 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:03:34.982019 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bc7ebe-9acd-46f6-8424-5dacbada4945-home" (OuterVolumeSpecName: "home") pod "68bc7ebe-9acd-46f6-8424-5dacbada4945" (UID: "68bc7ebe-9acd-46f6-8424-5dacbada4945"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 02:03:34.982113 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:03:34.982078 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68bc7ebe-9acd-46f6-8424-5dacbada4945-model-cache\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:03:34.983556 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:03:34.983533 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68bc7ebe-9acd-46f6-8424-5dacbada4945-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "68bc7ebe-9acd-46f6-8424-5dacbada4945" (UID: "68bc7ebe-9acd-46f6-8424-5dacbada4945"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 02:03:34.983678 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:03:34.983581 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bc7ebe-9acd-46f6-8424-5dacbada4945-dshm" (OuterVolumeSpecName: "dshm") pod "68bc7ebe-9acd-46f6-8424-5dacbada4945" (UID: "68bc7ebe-9acd-46f6-8424-5dacbada4945"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 02:03:34.983678 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:03:34.983666 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68bc7ebe-9acd-46f6-8424-5dacbada4945-kube-api-access-pgdpg" (OuterVolumeSpecName: "kube-api-access-pgdpg") pod "68bc7ebe-9acd-46f6-8424-5dacbada4945" (UID: "68bc7ebe-9acd-46f6-8424-5dacbada4945"). InnerVolumeSpecName "kube-api-access-pgdpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 02:03:35.082874 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:03:35.082842 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68bc7ebe-9acd-46f6-8424-5dacbada4945-tls-certs\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:03:35.082874 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:03:35.082869 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68bc7ebe-9acd-46f6-8424-5dacbada4945-home\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:03:35.082874 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:03:35.082879 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68bc7ebe-9acd-46f6-8424-5dacbada4945-tmp-dir\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:03:35.083125 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:03:35.082887 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68bc7ebe-9acd-46f6-8424-5dacbada4945-dshm\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:03:35.083125 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:03:35.082896 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgdpg\" (UniqueName: \"kubernetes.io/projected/68bc7ebe-9acd-46f6-8424-5dacbada4945-kube-api-access-pgdpg\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:03:35.146041 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:03:35.146011 2565 generic.go:356] "Generic (PLEG): container finished" podID="68bc7ebe-9acd-46f6-8424-5dacbada4945" containerID="c75765aeefa9aaf44cdcd0242117c04ab0628417e338b3862a7139c3e3fae04b" exitCode=0 Jul 28 02:03:35.146162 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:03:35.146088 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q" Jul 28 02:03:35.146162 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:03:35.146097 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q" event={"ID":"68bc7ebe-9acd-46f6-8424-5dacbada4945","Type":"ContainerDied","Data":"c75765aeefa9aaf44cdcd0242117c04ab0628417e338b3862a7139c3e3fae04b"} Jul 28 02:03:35.146162 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:03:35.146140 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q" event={"ID":"68bc7ebe-9acd-46f6-8424-5dacbada4945","Type":"ContainerDied","Data":"7f4d0563450752bcf4013d07362066038380feafc3e39dbd0e943f1b2be65a88"} Jul 28 02:03:35.146266 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:03:35.146162 2565 scope.go:117] "RemoveContainer" containerID="c75765aeefa9aaf44cdcd0242117c04ab0628417e338b3862a7139c3e3fae04b" Jul 28 02:03:35.154420 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:03:35.154402 2565 scope.go:117] "RemoveContainer" containerID="c75765aeefa9aaf44cdcd0242117c04ab0628417e338b3862a7139c3e3fae04b" Jul 28 02:03:35.154660 ip-10-0-131-23 kubenswrapper[2565]: E0728 02:03:35.154640 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c75765aeefa9aaf44cdcd0242117c04ab0628417e338b3862a7139c3e3fae04b\": container with ID starting with c75765aeefa9aaf44cdcd0242117c04ab0628417e338b3862a7139c3e3fae04b not found: ID does not exist" containerID="c75765aeefa9aaf44cdcd0242117c04ab0628417e338b3862a7139c3e3fae04b" Jul 28 02:03:35.154722 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:03:35.154668 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c75765aeefa9aaf44cdcd0242117c04ab0628417e338b3862a7139c3e3fae04b"} err="failed to get container status \"c75765aeefa9aaf44cdcd0242117c04ab0628417e338b3862a7139c3e3fae04b\": rpc error: code = NotFound desc = could not find container \"c75765aeefa9aaf44cdcd0242117c04ab0628417e338b3862a7139c3e3fae04b\": container with ID starting with c75765aeefa9aaf44cdcd0242117c04ab0628417e338b3862a7139c3e3fae04b not found: ID does not exist" Jul 28 02:03:35.171283 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:03:35.171260 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q"] Jul 28 02:03:35.175279 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:03:35.175258 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-de398f/sub-test-model-de398f-kserve-5fb758cdbf-cnv5q"] Jul 28 02:03:37.101481 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:03:37.101445 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68bc7ebe-9acd-46f6-8424-5dacbada4945" path="/var/lib/kubelet/pods/68bc7ebe-9acd-46f6-8424-5dacbada4945/volumes" Jul 28 02:04:13.801101 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:13.801070 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7"] Jul 28 02:04:13.801567 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:13.801416 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68bc7ebe-9acd-46f6-8424-5dacbada4945" containerName="main" Jul 28 02:04:13.801567 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:13.801426 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="68bc7ebe-9acd-46f6-8424-5dacbada4945" containerName="main" Jul 28 02:04:13.801567 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:13.801504 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="68bc7ebe-9acd-46f6-8424-5dacbada4945" containerName="main" Jul 28 02:04:13.804407 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:13.804386 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7" Jul 28 02:04:13.808776 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:13.808751 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-de398f\"/\"kube-root-ca.crt\"" Jul 28 02:04:13.810016 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:13.810000 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-de398f\"/\"default-dockercfg-wrrtm\"" Jul 28 02:04:13.810519 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:13.810501 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-de398f\"/\"rate-test-model-de398f-kserve-self-signed-certs\"" Jul 28 02:04:13.810616 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:13.810535 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-de398f\"/\"openshift-service-ca.crt\"" Jul 28 02:04:13.816125 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:13.816102 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7"] Jul 28 02:04:13.909492 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:13.909462 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73e975d4-4e32-4b20-8e1f-f7825cf99432-tmp-dir\") pod \"rate-test-model-de398f-kserve-8654d599f-snzx7\" (UID: \"73e975d4-4e32-4b20-8e1f-f7825cf99432\") " pod="ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7" Jul 28 02:04:13.909673 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:13.909501 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73e975d4-4e32-4b20-8e1f-f7825cf99432-dshm\") pod \"rate-test-model-de398f-kserve-8654d599f-snzx7\" (UID: \"73e975d4-4e32-4b20-8e1f-f7825cf99432\") " pod="ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7" Jul 28 02:04:13.909673 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:13.909525 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73e975d4-4e32-4b20-8e1f-f7825cf99432-home\") pod \"rate-test-model-de398f-kserve-8654d599f-snzx7\" (UID: \"73e975d4-4e32-4b20-8e1f-f7825cf99432\") " pod="ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7" Jul 28 02:04:13.909673 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:13.909539 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73e975d4-4e32-4b20-8e1f-f7825cf99432-model-cache\") pod \"rate-test-model-de398f-kserve-8654d599f-snzx7\" (UID: \"73e975d4-4e32-4b20-8e1f-f7825cf99432\") " pod="ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7" Jul 28 02:04:13.909673 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:13.909610 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73e975d4-4e32-4b20-8e1f-f7825cf99432-tls-certs\") pod \"rate-test-model-de398f-kserve-8654d599f-snzx7\" (UID: \"73e975d4-4e32-4b20-8e1f-f7825cf99432\") " pod="ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7" Jul 28 02:04:13.909673 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:13.909637 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz5r2\" (UniqueName: \"kubernetes.io/projected/73e975d4-4e32-4b20-8e1f-f7825cf99432-kube-api-access-mz5r2\") pod \"rate-test-model-de398f-kserve-8654d599f-snzx7\" (UID: \"73e975d4-4e32-4b20-8e1f-f7825cf99432\") " pod="ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7" Jul 28 02:04:14.010511 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:14.010475 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73e975d4-4e32-4b20-8e1f-f7825cf99432-tmp-dir\") pod \"rate-test-model-de398f-kserve-8654d599f-snzx7\" (UID: \"73e975d4-4e32-4b20-8e1f-f7825cf99432\") " pod="ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7" Jul 28 02:04:14.010511 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:14.010516 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73e975d4-4e32-4b20-8e1f-f7825cf99432-dshm\") pod \"rate-test-model-de398f-kserve-8654d599f-snzx7\" (UID: \"73e975d4-4e32-4b20-8e1f-f7825cf99432\") " pod="ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7" Jul 28 02:04:14.010756 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:14.010544 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73e975d4-4e32-4b20-8e1f-f7825cf99432-home\") pod \"rate-test-model-de398f-kserve-8654d599f-snzx7\" (UID: \"73e975d4-4e32-4b20-8e1f-f7825cf99432\") " pod="ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7" Jul 28 02:04:14.010756 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:14.010565 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73e975d4-4e32-4b20-8e1f-f7825cf99432-model-cache\") pod \"rate-test-model-de398f-kserve-8654d599f-snzx7\" (UID: \"73e975d4-4e32-4b20-8e1f-f7825cf99432\") " pod="ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7" Jul 28 02:04:14.010756 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:14.010635 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73e975d4-4e32-4b20-8e1f-f7825cf99432-tls-certs\") pod \"rate-test-model-de398f-kserve-8654d599f-snzx7\" (UID: \"73e975d4-4e32-4b20-8e1f-f7825cf99432\") " pod="ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7" Jul 28 02:04:14.010756 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:14.010659 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz5r2\" (UniqueName: \"kubernetes.io/projected/73e975d4-4e32-4b20-8e1f-f7825cf99432-kube-api-access-mz5r2\") pod \"rate-test-model-de398f-kserve-8654d599f-snzx7\" (UID: \"73e975d4-4e32-4b20-8e1f-f7825cf99432\") " pod="ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7" Jul 28 02:04:14.010967 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:14.010928 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73e975d4-4e32-4b20-8e1f-f7825cf99432-tmp-dir\") pod \"rate-test-model-de398f-kserve-8654d599f-snzx7\" (UID: \"73e975d4-4e32-4b20-8e1f-f7825cf99432\") " pod="ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7" Jul 28 02:04:14.011068 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:14.011019 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73e975d4-4e32-4b20-8e1f-f7825cf99432-home\") pod \"rate-test-model-de398f-kserve-8654d599f-snzx7\" (UID: \"73e975d4-4e32-4b20-8e1f-f7825cf99432\") " pod="ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7" Jul 28 02:04:14.011068 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:14.011063 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73e975d4-4e32-4b20-8e1f-f7825cf99432-model-cache\") pod \"rate-test-model-de398f-kserve-8654d599f-snzx7\" (UID: \"73e975d4-4e32-4b20-8e1f-f7825cf99432\") " pod="ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7" Jul 28 02:04:14.012917 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:14.012886 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73e975d4-4e32-4b20-8e1f-f7825cf99432-dshm\") pod \"rate-test-model-de398f-kserve-8654d599f-snzx7\" (UID: \"73e975d4-4e32-4b20-8e1f-f7825cf99432\") " pod="ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7" Jul 28 02:04:14.013028 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:14.013017 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73e975d4-4e32-4b20-8e1f-f7825cf99432-tls-certs\") pod \"rate-test-model-de398f-kserve-8654d599f-snzx7\" (UID: \"73e975d4-4e32-4b20-8e1f-f7825cf99432\") " pod="ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7" Jul 28 02:04:14.021819 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:14.021798 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz5r2\" (UniqueName: \"kubernetes.io/projected/73e975d4-4e32-4b20-8e1f-f7825cf99432-kube-api-access-mz5r2\") pod \"rate-test-model-de398f-kserve-8654d599f-snzx7\" (UID: \"73e975d4-4e32-4b20-8e1f-f7825cf99432\") " pod="ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7" Jul 28 02:04:14.115772 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:14.115697 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7" Jul 28 02:04:14.246862 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:14.246834 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7"] Jul 28 02:04:14.248705 ip-10-0-131-23 kubenswrapper[2565]: W0728 02:04:14.248676 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73e975d4_4e32_4b20_8e1f_f7825cf99432.slice/crio-03a2eb0fe8c579331cbbc90557ec01c8e5049eda67959aebe3358a6c29b49f3e WatchSource:0}: Error finding container 03a2eb0fe8c579331cbbc90557ec01c8e5049eda67959aebe3358a6c29b49f3e: Status 404 returned error can't find the container with id 03a2eb0fe8c579331cbbc90557ec01c8e5049eda67959aebe3358a6c29b49f3e Jul 28 02:04:14.337366 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:14.337339 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7" event={"ID":"73e975d4-4e32-4b20-8e1f-f7825cf99432","Type":"ContainerStarted","Data":"e64ee87aa5abe9a47862a361de10e4b50eea005c13c7b50f41187c2704e7d1f5"} Jul 28 02:04:14.337503 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:14.337378 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7" event={"ID":"73e975d4-4e32-4b20-8e1f-f7825cf99432","Type":"ContainerStarted","Data":"03a2eb0fe8c579331cbbc90557ec01c8e5049eda67959aebe3358a6c29b49f3e"} Jul 28 02:04:14.358920 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:14.358876 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7" podStartSLOduration=1.358861026 podStartE2EDuration="1.358861026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:04:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:04:14.355984445 +0000 UTC m=+2911.790382765" watchObservedRunningTime="2026-07-28 02:04:14.358861026 +0000 UTC m=+2911.793259351" Jul 28 02:04:24.116196 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:24.116146 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7" Jul 28 02:04:24.116720 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:24.116214 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7" Jul 28 02:04:24.128420 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:24.128392 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7" Jul 28 02:04:24.395838 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:04:24.395752 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7" Jul 28 02:05:24.199932 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:24.199851 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7"] Jul 28 02:05:39.218296 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:39.218228 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7" podUID="73e975d4-4e32-4b20-8e1f-f7825cf99432" containerName="main" containerID="cri-o://e64ee87aa5abe9a47862a361de10e4b50eea005c13c7b50f41187c2704e7d1f5" gracePeriod=45 Jul 28 02:05:39.464044 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:39.464002 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7" Jul 28 02:05:39.562584 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:39.562500 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73e975d4-4e32-4b20-8e1f-f7825cf99432-tmp-dir\") pod \"73e975d4-4e32-4b20-8e1f-f7825cf99432\" (UID: \"73e975d4-4e32-4b20-8e1f-f7825cf99432\") " Jul 28 02:05:39.562766 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:39.562597 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73e975d4-4e32-4b20-8e1f-f7825cf99432-dshm\") pod \"73e975d4-4e32-4b20-8e1f-f7825cf99432\" (UID: \"73e975d4-4e32-4b20-8e1f-f7825cf99432\") " Jul 28 02:05:39.562766 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:39.562620 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73e975d4-4e32-4b20-8e1f-f7825cf99432-model-cache\") pod \"73e975d4-4e32-4b20-8e1f-f7825cf99432\" (UID: \"73e975d4-4e32-4b20-8e1f-f7825cf99432\") " Jul 28 02:05:39.562766 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:39.562646 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mz5r2\" (UniqueName: \"kubernetes.io/projected/73e975d4-4e32-4b20-8e1f-f7825cf99432-kube-api-access-mz5r2\") pod \"73e975d4-4e32-4b20-8e1f-f7825cf99432\" (UID: \"73e975d4-4e32-4b20-8e1f-f7825cf99432\") " Jul 28 02:05:39.562766 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:39.562669 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73e975d4-4e32-4b20-8e1f-f7825cf99432-tls-certs\") pod \"73e975d4-4e32-4b20-8e1f-f7825cf99432\" (UID: \"73e975d4-4e32-4b20-8e1f-f7825cf99432\") " Jul 28 02:05:39.562766 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:39.562686 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73e975d4-4e32-4b20-8e1f-f7825cf99432-home\") pod \"73e975d4-4e32-4b20-8e1f-f7825cf99432\" (UID: \"73e975d4-4e32-4b20-8e1f-f7825cf99432\") " Jul 28 02:05:39.563081 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:39.562783 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e975d4-4e32-4b20-8e1f-f7825cf99432-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "73e975d4-4e32-4b20-8e1f-f7825cf99432" (UID: "73e975d4-4e32-4b20-8e1f-f7825cf99432"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 02:05:39.563081 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:39.562953 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73e975d4-4e32-4b20-8e1f-f7825cf99432-tmp-dir\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:05:39.563081 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:39.562928 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e975d4-4e32-4b20-8e1f-f7825cf99432-model-cache" (OuterVolumeSpecName: "model-cache") pod "73e975d4-4e32-4b20-8e1f-f7825cf99432" (UID: "73e975d4-4e32-4b20-8e1f-f7825cf99432"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 02:05:39.563081 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:39.563049 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e975d4-4e32-4b20-8e1f-f7825cf99432-home" (OuterVolumeSpecName: "home") pod "73e975d4-4e32-4b20-8e1f-f7825cf99432" (UID: "73e975d4-4e32-4b20-8e1f-f7825cf99432"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 02:05:39.564786 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:39.564756 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73e975d4-4e32-4b20-8e1f-f7825cf99432-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "73e975d4-4e32-4b20-8e1f-f7825cf99432" (UID: "73e975d4-4e32-4b20-8e1f-f7825cf99432"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 02:05:39.564895 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:39.564806 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e975d4-4e32-4b20-8e1f-f7825cf99432-kube-api-access-mz5r2" (OuterVolumeSpecName: "kube-api-access-mz5r2") pod "73e975d4-4e32-4b20-8e1f-f7825cf99432" (UID: "73e975d4-4e32-4b20-8e1f-f7825cf99432"). InnerVolumeSpecName "kube-api-access-mz5r2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 02:05:39.564960 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:39.564929 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e975d4-4e32-4b20-8e1f-f7825cf99432-dshm" (OuterVolumeSpecName: "dshm") pod "73e975d4-4e32-4b20-8e1f-f7825cf99432" (UID: "73e975d4-4e32-4b20-8e1f-f7825cf99432"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 02:05:39.664199 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:39.664164 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73e975d4-4e32-4b20-8e1f-f7825cf99432-dshm\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:05:39.664199 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:39.664192 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73e975d4-4e32-4b20-8e1f-f7825cf99432-model-cache\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:05:39.664199 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:39.664204 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz5r2\" (UniqueName: \"kubernetes.io/projected/73e975d4-4e32-4b20-8e1f-f7825cf99432-kube-api-access-mz5r2\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:05:39.664442 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:39.664215 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73e975d4-4e32-4b20-8e1f-f7825cf99432-tls-certs\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:05:39.664442 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:39.664224 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73e975d4-4e32-4b20-8e1f-f7825cf99432-home\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:05:39.768886 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:39.768853 2565 generic.go:356] "Generic (PLEG): container finished" podID="73e975d4-4e32-4b20-8e1f-f7825cf99432" containerID="e64ee87aa5abe9a47862a361de10e4b50eea005c13c7b50f41187c2704e7d1f5" exitCode=0 Jul 28 02:05:39.769080 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:39.768894 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7" Jul 28 02:05:39.769080 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:39.768924 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7" event={"ID":"73e975d4-4e32-4b20-8e1f-f7825cf99432","Type":"ContainerDied","Data":"e64ee87aa5abe9a47862a361de10e4b50eea005c13c7b50f41187c2704e7d1f5"} Jul 28 02:05:39.769080 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:39.768956 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7" event={"ID":"73e975d4-4e32-4b20-8e1f-f7825cf99432","Type":"ContainerDied","Data":"03a2eb0fe8c579331cbbc90557ec01c8e5049eda67959aebe3358a6c29b49f3e"} Jul 28 02:05:39.769080 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:39.768973 2565 scope.go:117] "RemoveContainer" containerID="e64ee87aa5abe9a47862a361de10e4b50eea005c13c7b50f41187c2704e7d1f5" Jul 28 02:05:39.778082 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:39.778069 2565 scope.go:117] "RemoveContainer" containerID="e64ee87aa5abe9a47862a361de10e4b50eea005c13c7b50f41187c2704e7d1f5" Jul 28 02:05:39.778329 ip-10-0-131-23 kubenswrapper[2565]: E0728 02:05:39.778308 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e64ee87aa5abe9a47862a361de10e4b50eea005c13c7b50f41187c2704e7d1f5\": container with ID starting with e64ee87aa5abe9a47862a361de10e4b50eea005c13c7b50f41187c2704e7d1f5 not found: ID does not exist" containerID="e64ee87aa5abe9a47862a361de10e4b50eea005c13c7b50f41187c2704e7d1f5" Jul 28 02:05:39.778388 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:39.778338 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e64ee87aa5abe9a47862a361de10e4b50eea005c13c7b50f41187c2704e7d1f5"} err="failed to get container status \"e64ee87aa5abe9a47862a361de10e4b50eea005c13c7b50f41187c2704e7d1f5\": rpc error: code = NotFound desc = could not find container \"e64ee87aa5abe9a47862a361de10e4b50eea005c13c7b50f41187c2704e7d1f5\": container with ID starting with e64ee87aa5abe9a47862a361de10e4b50eea005c13c7b50f41187c2704e7d1f5 not found: ID does not exist" Jul 28 02:05:39.791569 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:39.791536 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7"] Jul 28 02:05:39.798000 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:39.797978 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-de398f/rate-test-model-de398f-kserve-8654d599f-snzx7"] Jul 28 02:05:41.101267 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:41.101230 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73e975d4-4e32-4b20-8e1f-f7825cf99432" path="/var/lib/kubelet/pods/73e975d4-4e32-4b20-8e1f-f7825cf99432/volumes" Jul 28 02:05:46.923450 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:46.923410 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-49ecf4-84d95869cb-vkddb"] Jul 28 02:05:46.923894 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:46.923775 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73e975d4-4e32-4b20-8e1f-f7825cf99432" containerName="main" Jul 28 02:05:46.923894 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:46.923786 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="73e975d4-4e32-4b20-8e1f-f7825cf99432" containerName="main" Jul 28 02:05:46.923894 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:46.923838 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="73e975d4-4e32-4b20-8e1f-f7825cf99432" containerName="main" Jul 28 02:05:46.928218 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:46.928199 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49ecf4-84d95869cb-vkddb" Jul 28 02:05:46.930921 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:46.930903 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-49ecf4-serving-cert\"" Jul 28 02:05:46.936968 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:46.936943 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-49ecf4-84d95869cb-vkddb"] Jul 28 02:05:47.029509 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:47.029475 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h4rc\" (UniqueName: \"kubernetes.io/projected/fc0ceaac-b443-4868-811e-60201f68c659-kube-api-access-9h4rc\") pod \"maas-api-e2e-ipp-a-49ecf4-84d95869cb-vkddb\" (UID: \"fc0ceaac-b443-4868-811e-60201f68c659\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49ecf4-84d95869cb-vkddb" Jul 28 02:05:47.029686 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:47.029620 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc0ceaac-b443-4868-811e-60201f68c659-maas-api-tls\") pod \"maas-api-e2e-ipp-a-49ecf4-84d95869cb-vkddb\" (UID: \"fc0ceaac-b443-4868-811e-60201f68c659\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49ecf4-84d95869cb-vkddb" Jul 28 02:05:47.029686 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:47.029642 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc0ceaac-b443-4868-811e-60201f68c659-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-49ecf4-84d95869cb-vkddb\" (UID: \"fc0ceaac-b443-4868-811e-60201f68c659\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49ecf4-84d95869cb-vkddb" Jul 28 02:05:47.131043 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:47.131009 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc0ceaac-b443-4868-811e-60201f68c659-maas-api-tls\") pod \"maas-api-e2e-ipp-a-49ecf4-84d95869cb-vkddb\" (UID: \"fc0ceaac-b443-4868-811e-60201f68c659\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49ecf4-84d95869cb-vkddb" Jul 28 02:05:47.131231 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:47.131065 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc0ceaac-b443-4868-811e-60201f68c659-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-49ecf4-84d95869cb-vkddb\" (UID: \"fc0ceaac-b443-4868-811e-60201f68c659\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49ecf4-84d95869cb-vkddb" Jul 28 02:05:47.131231 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:47.131097 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h4rc\" (UniqueName: \"kubernetes.io/projected/fc0ceaac-b443-4868-811e-60201f68c659-kube-api-access-9h4rc\") pod \"maas-api-e2e-ipp-a-49ecf4-84d95869cb-vkddb\" (UID: \"fc0ceaac-b443-4868-811e-60201f68c659\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49ecf4-84d95869cb-vkddb" Jul 28 02:05:47.131824 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:47.131801 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc0ceaac-b443-4868-811e-60201f68c659-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-49ecf4-84d95869cb-vkddb\" (UID: \"fc0ceaac-b443-4868-811e-60201f68c659\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49ecf4-84d95869cb-vkddb" Jul 28 02:05:47.133595 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:47.133569 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc0ceaac-b443-4868-811e-60201f68c659-maas-api-tls\") pod \"maas-api-e2e-ipp-a-49ecf4-84d95869cb-vkddb\" (UID: \"fc0ceaac-b443-4868-811e-60201f68c659\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49ecf4-84d95869cb-vkddb" Jul 28 02:05:47.152643 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:47.152615 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h4rc\" (UniqueName: \"kubernetes.io/projected/fc0ceaac-b443-4868-811e-60201f68c659-kube-api-access-9h4rc\") pod \"maas-api-e2e-ipp-a-49ecf4-84d95869cb-vkddb\" (UID: \"fc0ceaac-b443-4868-811e-60201f68c659\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49ecf4-84d95869cb-vkddb" Jul 28 02:05:47.239706 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:47.239670 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49ecf4-84d95869cb-vkddb" Jul 28 02:05:47.381024 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:47.380997 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-49ecf4-84d95869cb-vkddb"] Jul 28 02:05:47.383527 ip-10-0-131-23 kubenswrapper[2565]: W0728 02:05:47.383498 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc0ceaac_b443_4868_811e_60201f68c659.slice/crio-1e8904ef2cc39ea85a45582f48be3ee7fbdd1d4a347858757b18baa0fabf7ed3 WatchSource:0}: Error finding container 1e8904ef2cc39ea85a45582f48be3ee7fbdd1d4a347858757b18baa0fabf7ed3: Status 404 returned error can't find the container with id 1e8904ef2cc39ea85a45582f48be3ee7fbdd1d4a347858757b18baa0fabf7ed3 Jul 28 02:05:47.385011 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:47.384992 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 02:05:47.819052 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:47.819001 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49ecf4-84d95869cb-vkddb" event={"ID":"fc0ceaac-b443-4868-811e-60201f68c659","Type":"ContainerStarted","Data":"1e8904ef2cc39ea85a45582f48be3ee7fbdd1d4a347858757b18baa0fabf7ed3"} Jul 28 02:05:48.824778 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:48.824741 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49ecf4-84d95869cb-vkddb" event={"ID":"fc0ceaac-b443-4868-811e-60201f68c659","Type":"ContainerStarted","Data":"eb83e9d35d43e903643a8c41fcd1bf409c9b0ce17ae0fd5317b498231b772e78"} Jul 28 02:05:48.825196 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:48.824825 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49ecf4-84d95869cb-vkddb" Jul 28 02:05:48.843380 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:48.843333 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49ecf4-84d95869cb-vkddb" podStartSLOduration=2.390727148 podStartE2EDuration="2.843315056s" totalImagesPullingTime="452.587908ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:05:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:05:48.842601511 +0000 UTC m=+3006.276999826" watchObservedRunningTime="2026-07-28 02:05:48.843315056 +0000 UTC m=+3006.277713371" Jul 28 02:05:54.835453 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:05:54.835424 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49ecf4-84d95869cb-vkddb" Jul 28 02:06:05.970597 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:05.970565 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k"] Jul 28 02:06:05.973952 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:05.973932 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k" Jul 28 02:06:05.976862 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:05.976841 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-e0a963-serving-cert\"" Jul 28 02:06:05.989225 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:05.989193 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k"] Jul 28 02:06:06.102406 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:06.102373 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45abba4e-fffa-4e78-b46b-e3367d8c6816-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k\" (UID: \"45abba4e-fffa-4e78-b46b-e3367d8c6816\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k" Jul 28 02:06:06.102589 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:06.102439 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gktck\" (UniqueName: \"kubernetes.io/projected/45abba4e-fffa-4e78-b46b-e3367d8c6816-kube-api-access-gktck\") pod \"maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k\" (UID: \"45abba4e-fffa-4e78-b46b-e3367d8c6816\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k" Jul 28 02:06:06.102589 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:06.102555 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45abba4e-fffa-4e78-b46b-e3367d8c6816-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k\" (UID: \"45abba4e-fffa-4e78-b46b-e3367d8c6816\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k" Jul 28 02:06:06.203127 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:06.203093 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gktck\" (UniqueName: \"kubernetes.io/projected/45abba4e-fffa-4e78-b46b-e3367d8c6816-kube-api-access-gktck\") pod \"maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k\" (UID: \"45abba4e-fffa-4e78-b46b-e3367d8c6816\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k" Jul 28 02:06:06.203298 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:06.203164 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45abba4e-fffa-4e78-b46b-e3367d8c6816-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k\" (UID: \"45abba4e-fffa-4e78-b46b-e3367d8c6816\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k" Jul 28 02:06:06.203298 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:06.203195 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45abba4e-fffa-4e78-b46b-e3367d8c6816-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k\" (UID: \"45abba4e-fffa-4e78-b46b-e3367d8c6816\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k" Jul 28 02:06:06.203298 ip-10-0-131-23 kubenswrapper[2565]: E0728 02:06:06.203295 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a963-serving-cert: secret "maas-api-e2e-ipp-b-e0a963-serving-cert" not found Jul 28 02:06:06.203411 ip-10-0-131-23 kubenswrapper[2565]: E0728 02:06:06.203349 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45abba4e-fffa-4e78-b46b-e3367d8c6816-maas-api-tls podName:45abba4e-fffa-4e78-b46b-e3367d8c6816 nodeName:}" failed. No retries permitted until 2026-07-28 02:06:06.703332601 +0000 UTC m=+3024.137730893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/45abba4e-fffa-4e78-b46b-e3367d8c6816-maas-api-tls") pod "maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k" (UID: "45abba4e-fffa-4e78-b46b-e3367d8c6816") : secret "maas-api-e2e-ipp-b-e0a963-serving-cert" not found Jul 28 02:06:06.203829 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:06.203808 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45abba4e-fffa-4e78-b46b-e3367d8c6816-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k\" (UID: \"45abba4e-fffa-4e78-b46b-e3367d8c6816\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k" Jul 28 02:06:06.215473 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:06.215445 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gktck\" (UniqueName: \"kubernetes.io/projected/45abba4e-fffa-4e78-b46b-e3367d8c6816-kube-api-access-gktck\") pod \"maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k\" (UID: \"45abba4e-fffa-4e78-b46b-e3367d8c6816\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k" Jul 28 02:06:06.710138 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:06.710102 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45abba4e-fffa-4e78-b46b-e3367d8c6816-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k\" (UID: \"45abba4e-fffa-4e78-b46b-e3367d8c6816\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k" Jul 28 02:06:06.712639 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:06.712614 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45abba4e-fffa-4e78-b46b-e3367d8c6816-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k\" (UID: \"45abba4e-fffa-4e78-b46b-e3367d8c6816\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k" Jul 28 02:06:06.887743 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:06.887706 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k" Jul 28 02:06:07.041441 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:07.041413 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k"] Jul 28 02:06:07.043227 ip-10-0-131-23 kubenswrapper[2565]: W0728 02:06:07.043200 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45abba4e_fffa_4e78_b46b_e3367d8c6816.slice/crio-1966d3892ef13658d4afb7347c4aee7f537b99596fa90e56667ed85a79150226 WatchSource:0}: Error finding container 1966d3892ef13658d4afb7347c4aee7f537b99596fa90e56667ed85a79150226: Status 404 returned error can't find the container with id 1966d3892ef13658d4afb7347c4aee7f537b99596fa90e56667ed85a79150226 Jul 28 02:06:07.924860 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:07.924814 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k" event={"ID":"45abba4e-fffa-4e78-b46b-e3367d8c6816","Type":"ContainerStarted","Data":"dbdc7e1f3ab52223eba9da559a6bcd049e5d5d237e6586d79be3d119a572a33c"} Jul 28 02:06:07.925042 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:07.924883 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k" Jul 28 02:06:07.925042 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:07.924905 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k" event={"ID":"45abba4e-fffa-4e78-b46b-e3367d8c6816","Type":"ContainerStarted","Data":"1966d3892ef13658d4afb7347c4aee7f537b99596fa90e56667ed85a79150226"} Jul 28 02:06:07.942487 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:07.942438 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k" podStartSLOduration=2.192959561 podStartE2EDuration="2.942422516s" totalImagesPullingTime="749.462955ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:06:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:06:07.940340262 +0000 UTC m=+3025.374738580" watchObservedRunningTime="2026-07-28 02:06:07.942422516 +0000 UTC m=+3025.376820830" Jul 28 02:06:13.935246 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:13.935213 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k" Jul 28 02:06:35.002812 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:35.002718 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv"] Jul 28 02:06:35.006416 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:35.006388 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv" Jul 28 02:06:35.010395 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:35.010364 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-49ecf4\"/\"kube-root-ca.crt\"" Jul 28 02:06:35.010543 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:35.010414 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-49ecf4\"/\"openshift-service-ca.crt\"" Jul 28 02:06:35.010543 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:35.010448 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-49ecf4\"/\"ipp-route-49ecf4-kserve-self-signed-certs\"" Jul 28 02:06:35.010543 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:35.010416 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-49ecf4\"/\"default-dockercfg-vqffb\"" Jul 28 02:06:35.016777 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:35.016743 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv"] Jul 28 02:06:35.054866 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:35.054826 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2b2286b-40ac-41b0-9aac-64ed107953e8-model-cache\") pod \"ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv\" (UID: \"a2b2286b-40ac-41b0-9aac-64ed107953e8\") " pod="ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv" Jul 28 02:06:35.054866 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:35.054863 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2b2286b-40ac-41b0-9aac-64ed107953e8-home\") pod \"ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv\" (UID: \"a2b2286b-40ac-41b0-9aac-64ed107953e8\") " pod="ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv" Jul 28 02:06:35.055149 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:35.054984 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn8qt\" (UniqueName: \"kubernetes.io/projected/a2b2286b-40ac-41b0-9aac-64ed107953e8-kube-api-access-kn8qt\") pod \"ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv\" (UID: \"a2b2286b-40ac-41b0-9aac-64ed107953e8\") " pod="ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv" Jul 28 02:06:35.055149 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:35.055059 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2b2286b-40ac-41b0-9aac-64ed107953e8-tls-certs\") pod \"ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv\" (UID: \"a2b2286b-40ac-41b0-9aac-64ed107953e8\") " pod="ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv" Jul 28 02:06:35.055149 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:35.055121 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2b2286b-40ac-41b0-9aac-64ed107953e8-dshm\") pod \"ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv\" (UID: \"a2b2286b-40ac-41b0-9aac-64ed107953e8\") " pod="ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv" Jul 28 02:06:35.055321 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:35.055151 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2b2286b-40ac-41b0-9aac-64ed107953e8-tmp-dir\") pod \"ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv\" (UID: \"a2b2286b-40ac-41b0-9aac-64ed107953e8\") " pod="ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv" Jul 28 02:06:35.155807 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:35.155769 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn8qt\" (UniqueName: \"kubernetes.io/projected/a2b2286b-40ac-41b0-9aac-64ed107953e8-kube-api-access-kn8qt\") pod \"ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv\" (UID: \"a2b2286b-40ac-41b0-9aac-64ed107953e8\") " pod="ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv" Jul 28 02:06:35.155807 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:35.155812 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2b2286b-40ac-41b0-9aac-64ed107953e8-tls-certs\") pod \"ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv\" (UID: \"a2b2286b-40ac-41b0-9aac-64ed107953e8\") " pod="ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv" Jul 28 02:06:35.156112 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:35.155877 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2b2286b-40ac-41b0-9aac-64ed107953e8-dshm\") pod \"ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv\" (UID: \"a2b2286b-40ac-41b0-9aac-64ed107953e8\") " pod="ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv" Jul 28 02:06:35.156112 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:35.155915 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2b2286b-40ac-41b0-9aac-64ed107953e8-tmp-dir\") pod \"ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv\" (UID: \"a2b2286b-40ac-41b0-9aac-64ed107953e8\") " pod="ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv" Jul 28 02:06:35.156112 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:35.156098 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2b2286b-40ac-41b0-9aac-64ed107953e8-model-cache\") pod \"ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv\" (UID: \"a2b2286b-40ac-41b0-9aac-64ed107953e8\") " pod="ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv" Jul 28 02:06:35.156287 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:35.156133 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2b2286b-40ac-41b0-9aac-64ed107953e8-home\") pod \"ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv\" (UID: \"a2b2286b-40ac-41b0-9aac-64ed107953e8\") " pod="ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv" Jul 28 02:06:35.156394 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:35.156373 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2b2286b-40ac-41b0-9aac-64ed107953e8-tmp-dir\") pod \"ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv\" (UID: \"a2b2286b-40ac-41b0-9aac-64ed107953e8\") " pod="ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv" Jul 28 02:06:35.156444 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:35.156429 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2b2286b-40ac-41b0-9aac-64ed107953e8-model-cache\") pod \"ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv\" (UID: \"a2b2286b-40ac-41b0-9aac-64ed107953e8\") " pod="ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv" Jul 28 02:06:35.156547 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:35.156530 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2b2286b-40ac-41b0-9aac-64ed107953e8-home\") pod \"ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv\" (UID: \"a2b2286b-40ac-41b0-9aac-64ed107953e8\") " pod="ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv" Jul 28 02:06:35.158196 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:35.158175 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2b2286b-40ac-41b0-9aac-64ed107953e8-dshm\") pod \"ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv\" (UID: \"a2b2286b-40ac-41b0-9aac-64ed107953e8\") " pod="ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv" Jul 28 02:06:35.158371 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:35.158356 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2b2286b-40ac-41b0-9aac-64ed107953e8-tls-certs\") pod \"ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv\" (UID: \"a2b2286b-40ac-41b0-9aac-64ed107953e8\") " pod="ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv" Jul 28 02:06:35.165355 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:35.165330 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn8qt\" (UniqueName: \"kubernetes.io/projected/a2b2286b-40ac-41b0-9aac-64ed107953e8-kube-api-access-kn8qt\") pod \"ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv\" (UID: \"a2b2286b-40ac-41b0-9aac-64ed107953e8\") " pod="ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv" Jul 28 02:06:35.317571 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:35.317469 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv" Jul 28 02:06:35.457286 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:35.457201 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv"] Jul 28 02:06:35.459648 ip-10-0-131-23 kubenswrapper[2565]: W0728 02:06:35.459619 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2b2286b_40ac_41b0_9aac_64ed107953e8.slice/crio-a35bc628196937d1fbffbe0312d4a0aedbc66bfe80427a281adee66f9af25ca4 WatchSource:0}: Error finding container a35bc628196937d1fbffbe0312d4a0aedbc66bfe80427a281adee66f9af25ca4: Status 404 returned error can't find the container with id a35bc628196937d1fbffbe0312d4a0aedbc66bfe80427a281adee66f9af25ca4 Jul 28 02:06:36.072631 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:36.072589 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv" event={"ID":"a2b2286b-40ac-41b0-9aac-64ed107953e8","Type":"ContainerStarted","Data":"f0c23497381a286ea7c77f2473a91469ad5c44c4b163c433325a13ae709b38e1"} Jul 28 02:06:36.072631 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:36.072637 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv" event={"ID":"a2b2286b-40ac-41b0-9aac-64ed107953e8","Type":"ContainerStarted","Data":"a35bc628196937d1fbffbe0312d4a0aedbc66bfe80427a281adee66f9af25ca4"} Jul 28 02:06:36.088510 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:36.088458 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv" podStartSLOduration=2.088438911 podStartE2EDuration="2.088438911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:06:36.088155777 +0000 UTC m=+3053.522554092" watchObservedRunningTime="2026-07-28 02:06:36.088438911 +0000 UTC m=+3053.522837225" Jul 28 02:06:45.318390 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:45.318345 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv" Jul 28 02:06:45.320771 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:45.318404 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv" Jul 28 02:06:45.330977 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:45.330954 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv" Jul 28 02:06:46.132644 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:06:46.132613 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv" Jul 28 02:07:24.982793 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:24.982754 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw"] Jul 28 02:07:24.987185 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:24.987156 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw" Jul 28 02:07:24.991156 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:24.991126 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-e78415-serving-cert\"" Jul 28 02:07:25.000165 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:25.000133 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw"] Jul 28 02:07:25.119453 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:25.119421 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03fb507c-6657-4b97-b7d8-75ed80a61593-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw\" (UID: \"03fb507c-6657-4b97-b7d8-75ed80a61593\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw" Jul 28 02:07:25.119657 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:25.119468 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp8zk\" (UniqueName: \"kubernetes.io/projected/03fb507c-6657-4b97-b7d8-75ed80a61593-kube-api-access-xp8zk\") pod \"maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw\" (UID: \"03fb507c-6657-4b97-b7d8-75ed80a61593\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw" Jul 28 02:07:25.119657 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:25.119508 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03fb507c-6657-4b97-b7d8-75ed80a61593-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw\" (UID: \"03fb507c-6657-4b97-b7d8-75ed80a61593\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw" Jul 28 02:07:25.220639 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:25.220595 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03fb507c-6657-4b97-b7d8-75ed80a61593-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw\" (UID: \"03fb507c-6657-4b97-b7d8-75ed80a61593\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw" Jul 28 02:07:25.220639 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:25.220643 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp8zk\" (UniqueName: \"kubernetes.io/projected/03fb507c-6657-4b97-b7d8-75ed80a61593-kube-api-access-xp8zk\") pod \"maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw\" (UID: \"03fb507c-6657-4b97-b7d8-75ed80a61593\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw" Jul 28 02:07:25.220863 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:25.220668 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03fb507c-6657-4b97-b7d8-75ed80a61593-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw\" (UID: \"03fb507c-6657-4b97-b7d8-75ed80a61593\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw" Jul 28 02:07:25.220863 ip-10-0-131-23 kubenswrapper[2565]: E0728 02:07:25.220853 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e78415-serving-cert: secret "maas-api-e2e-ipp-cleanup-e78415-serving-cert" not found Jul 28 02:07:25.220938 ip-10-0-131-23 kubenswrapper[2565]: E0728 02:07:25.220928 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03fb507c-6657-4b97-b7d8-75ed80a61593-maas-api-tls podName:03fb507c-6657-4b97-b7d8-75ed80a61593 nodeName:}" failed. No retries permitted until 2026-07-28 02:07:25.720898624 +0000 UTC m=+3103.155296918 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/03fb507c-6657-4b97-b7d8-75ed80a61593-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw" (UID: "03fb507c-6657-4b97-b7d8-75ed80a61593") : secret "maas-api-e2e-ipp-cleanup-e78415-serving-cert" not found Jul 28 02:07:25.221366 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:25.221347 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03fb507c-6657-4b97-b7d8-75ed80a61593-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw\" (UID: \"03fb507c-6657-4b97-b7d8-75ed80a61593\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw" Jul 28 02:07:25.232329 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:25.232295 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp8zk\" (UniqueName: \"kubernetes.io/projected/03fb507c-6657-4b97-b7d8-75ed80a61593-kube-api-access-xp8zk\") pod \"maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw\" (UID: \"03fb507c-6657-4b97-b7d8-75ed80a61593\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw" Jul 28 02:07:25.726344 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:25.726302 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03fb507c-6657-4b97-b7d8-75ed80a61593-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw\" (UID: \"03fb507c-6657-4b97-b7d8-75ed80a61593\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw" Jul 28 02:07:25.729516 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:25.729482 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03fb507c-6657-4b97-b7d8-75ed80a61593-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw\" (UID: \"03fb507c-6657-4b97-b7d8-75ed80a61593\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw" Jul 28 02:07:25.905454 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:25.905409 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw" Jul 28 02:07:26.040401 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:26.040376 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw"] Jul 28 02:07:26.042006 ip-10-0-131-23 kubenswrapper[2565]: W0728 02:07:26.041967 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03fb507c_6657_4b97_b7d8_75ed80a61593.slice/crio-1a81e56d370a9b173f13b89a16017dce09a106cc481ff826c611916d7151dac1 WatchSource:0}: Error finding container 1a81e56d370a9b173f13b89a16017dce09a106cc481ff826c611916d7151dac1: Status 404 returned error can't find the container with id 1a81e56d370a9b173f13b89a16017dce09a106cc481ff826c611916d7151dac1 Jul 28 02:07:26.339836 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:26.339743 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw" event={"ID":"03fb507c-6657-4b97-b7d8-75ed80a61593","Type":"ContainerStarted","Data":"1a81e56d370a9b173f13b89a16017dce09a106cc481ff826c611916d7151dac1"} Jul 28 02:07:27.347212 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:27.347174 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw" event={"ID":"03fb507c-6657-4b97-b7d8-75ed80a61593","Type":"ContainerStarted","Data":"b1aa1c23b04ef44827741adf3f725d0e6a76be7572c7782c79f9b84caa2e200c"} Jul 28 02:07:27.347629 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:27.347294 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw" Jul 28 02:07:27.365980 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:27.365924 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw" podStartSLOduration=2.979947703 podStartE2EDuration="3.365905543s" totalImagesPullingTime="385.95784ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:07:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:07:27.364767812 +0000 UTC m=+3104.799166125" watchObservedRunningTime="2026-07-28 02:07:27.365905543 +0000 UTC m=+3104.800303916" Jul 28 02:07:33.358659 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:33.358578 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw" Jul 28 02:07:43.809871 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:43.809836 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e78415-5hzkc"] Jul 28 02:07:43.813468 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:43.813449 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e78415-5hzkc" Jul 28 02:07:43.815975 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:43.815947 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wjxvt\"" Jul 28 02:07:43.821953 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:43.821311 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e78415-5hzkc"] Jul 28 02:07:43.905220 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:43.905185 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40587140-7a6d-4151-bcbc-44f583ca6d36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-e78415-5hzkc\" (UID: \"40587140-7a6d-4151-bcbc-44f583ca6d36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e78415-5hzkc" Jul 28 02:07:44.005921 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:44.005882 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40587140-7a6d-4151-bcbc-44f583ca6d36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-e78415-5hzkc\" (UID: \"40587140-7a6d-4151-bcbc-44f583ca6d36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e78415-5hzkc" Jul 28 02:07:44.006555 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:44.006534 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40587140-7a6d-4151-bcbc-44f583ca6d36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-e78415-5hzkc\" (UID: \"40587140-7a6d-4151-bcbc-44f583ca6d36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e78415-5hzkc" Jul 28 02:07:44.125989 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:44.125894 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e78415-5hzkc" Jul 28 02:07:44.261706 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:44.261679 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e78415-5hzkc"] Jul 28 02:07:44.263192 ip-10-0-131-23 kubenswrapper[2565]: W0728 02:07:44.263164 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40587140_7a6d_4151_bcbc_44f583ca6d36.slice/crio-df795e2893b669ad5d263ca64926a80e9195e2fabc5ac85c09f901e342285037 WatchSource:0}: Error finding container df795e2893b669ad5d263ca64926a80e9195e2fabc5ac85c09f901e342285037: Status 404 returned error can't find the container with id df795e2893b669ad5d263ca64926a80e9195e2fabc5ac85c09f901e342285037 Jul 28 02:07:44.439603 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:44.439569 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e78415-5hzkc" event={"ID":"40587140-7a6d-4151-bcbc-44f583ca6d36","Type":"ContainerStarted","Data":"df795e2893b669ad5d263ca64926a80e9195e2fabc5ac85c09f901e342285037"} Jul 28 02:07:45.445983 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:45.445949 2565 generic.go:356] "Generic (PLEG): container finished" podID="40587140-7a6d-4151-bcbc-44f583ca6d36" containerID="4fb12e6b9cba208712aafdf8e71e2387d0c3072f8b898445d8a2942f2a3d3f10" exitCode=0 Jul 28 02:07:45.446394 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:45.446005 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e78415-5hzkc" event={"ID":"40587140-7a6d-4151-bcbc-44f583ca6d36","Type":"ContainerDied","Data":"4fb12e6b9cba208712aafdf8e71e2387d0c3072f8b898445d8a2942f2a3d3f10"} Jul 28 02:07:46.583315 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:46.583288 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e78415-5hzkc" Jul 28 02:07:46.731205 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:46.731110 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40587140-7a6d-4151-bcbc-44f583ca6d36-maas-api-service-ca\") pod \"40587140-7a6d-4151-bcbc-44f583ca6d36\" (UID: \"40587140-7a6d-4151-bcbc-44f583ca6d36\") " Jul 28 02:07:46.731492 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:46.731469 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40587140-7a6d-4151-bcbc-44f583ca6d36-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "40587140-7a6d-4151-bcbc-44f583ca6d36" (UID: "40587140-7a6d-4151-bcbc-44f583ca6d36"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 02:07:46.832777 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:46.832736 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40587140-7a6d-4151-bcbc-44f583ca6d36-maas-api-service-ca\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:07:47.458023 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:47.457987 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e78415-5hzkc" Jul 28 02:07:47.458231 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:47.458059 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e78415-5hzkc" event={"ID":"40587140-7a6d-4151-bcbc-44f583ca6d36","Type":"ContainerDied","Data":"df795e2893b669ad5d263ca64926a80e9195e2fabc5ac85c09f901e342285037"} Jul 28 02:07:47.458231 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:47.458105 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df795e2893b669ad5d263ca64926a80e9195e2fabc5ac85c09f901e342285037" Jul 28 02:07:55.962433 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:55.962401 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw"] Jul 28 02:07:55.963017 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:55.962680 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw" podUID="03fb507c-6657-4b97-b7d8-75ed80a61593" containerName="maas-api" containerID="cri-o://b1aa1c23b04ef44827741adf3f725d0e6a76be7572c7782c79f9b84caa2e200c" gracePeriod=30 Jul 28 02:07:56.230649 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:56.230584 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw" Jul 28 02:07:56.322409 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:56.322373 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xp8zk\" (UniqueName: \"kubernetes.io/projected/03fb507c-6657-4b97-b7d8-75ed80a61593-kube-api-access-xp8zk\") pod \"03fb507c-6657-4b97-b7d8-75ed80a61593\" (UID: \"03fb507c-6657-4b97-b7d8-75ed80a61593\") " Jul 28 02:07:56.322621 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:56.322431 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03fb507c-6657-4b97-b7d8-75ed80a61593-openshift-service-ca\") pod \"03fb507c-6657-4b97-b7d8-75ed80a61593\" (UID: \"03fb507c-6657-4b97-b7d8-75ed80a61593\") " Jul 28 02:07:56.322621 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:56.322465 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03fb507c-6657-4b97-b7d8-75ed80a61593-maas-api-tls\") pod \"03fb507c-6657-4b97-b7d8-75ed80a61593\" (UID: \"03fb507c-6657-4b97-b7d8-75ed80a61593\") " Jul 28 02:07:56.322875 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:56.322851 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03fb507c-6657-4b97-b7d8-75ed80a61593-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "03fb507c-6657-4b97-b7d8-75ed80a61593" (UID: "03fb507c-6657-4b97-b7d8-75ed80a61593"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 02:07:56.324600 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:56.324572 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03fb507c-6657-4b97-b7d8-75ed80a61593-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "03fb507c-6657-4b97-b7d8-75ed80a61593" (UID: "03fb507c-6657-4b97-b7d8-75ed80a61593"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 02:07:56.324703 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:56.324635 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03fb507c-6657-4b97-b7d8-75ed80a61593-kube-api-access-xp8zk" (OuterVolumeSpecName: "kube-api-access-xp8zk") pod "03fb507c-6657-4b97-b7d8-75ed80a61593" (UID: "03fb507c-6657-4b97-b7d8-75ed80a61593"). InnerVolumeSpecName "kube-api-access-xp8zk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 02:07:56.423701 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:56.423662 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp8zk\" (UniqueName: \"kubernetes.io/projected/03fb507c-6657-4b97-b7d8-75ed80a61593-kube-api-access-xp8zk\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:07:56.423701 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:56.423693 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03fb507c-6657-4b97-b7d8-75ed80a61593-openshift-service-ca\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:07:56.423701 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:56.423703 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03fb507c-6657-4b97-b7d8-75ed80a61593-maas-api-tls\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:07:56.506063 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:56.505941 2565 generic.go:356] "Generic (PLEG): container finished" podID="03fb507c-6657-4b97-b7d8-75ed80a61593" containerID="b1aa1c23b04ef44827741adf3f725d0e6a76be7572c7782c79f9b84caa2e200c" exitCode=0 Jul 28 02:07:56.506063 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:56.505991 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw" Jul 28 02:07:56.506063 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:56.506043 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw" event={"ID":"03fb507c-6657-4b97-b7d8-75ed80a61593","Type":"ContainerDied","Data":"b1aa1c23b04ef44827741adf3f725d0e6a76be7572c7782c79f9b84caa2e200c"} Jul 28 02:07:56.506329 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:56.506105 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw" event={"ID":"03fb507c-6657-4b97-b7d8-75ed80a61593","Type":"ContainerDied","Data":"1a81e56d370a9b173f13b89a16017dce09a106cc481ff826c611916d7151dac1"} Jul 28 02:07:56.506329 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:56.506140 2565 scope.go:117] "RemoveContainer" containerID="b1aa1c23b04ef44827741adf3f725d0e6a76be7572c7782c79f9b84caa2e200c" Jul 28 02:07:56.515542 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:56.515486 2565 scope.go:117] "RemoveContainer" containerID="b1aa1c23b04ef44827741adf3f725d0e6a76be7572c7782c79f9b84caa2e200c" Jul 28 02:07:56.515807 ip-10-0-131-23 kubenswrapper[2565]: E0728 02:07:56.515785 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1aa1c23b04ef44827741adf3f725d0e6a76be7572c7782c79f9b84caa2e200c\": container with ID starting with b1aa1c23b04ef44827741adf3f725d0e6a76be7572c7782c79f9b84caa2e200c not found: ID does not exist" containerID="b1aa1c23b04ef44827741adf3f725d0e6a76be7572c7782c79f9b84caa2e200c" Jul 28 02:07:56.515898 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:56.515816 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1aa1c23b04ef44827741adf3f725d0e6a76be7572c7782c79f9b84caa2e200c"} err="failed to get container status \"b1aa1c23b04ef44827741adf3f725d0e6a76be7572c7782c79f9b84caa2e200c\": rpc error: code = NotFound desc = could not find container \"b1aa1c23b04ef44827741adf3f725d0e6a76be7572c7782c79f9b84caa2e200c\": container with ID starting with b1aa1c23b04ef44827741adf3f725d0e6a76be7572c7782c79f9b84caa2e200c not found: ID does not exist" Jul 28 02:07:56.528360 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:56.528333 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw"] Jul 28 02:07:56.532164 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:56.532141 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e78415-6957f57686-rx7xw"] Jul 28 02:07:57.101595 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:57.101562 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03fb507c-6657-4b97-b7d8-75ed80a61593" path="/var/lib/kubelet/pods/03fb507c-6657-4b97-b7d8-75ed80a61593/volumes" Jul 28 02:07:58.892712 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:58.892666 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e78415-5hzkc"] Jul 28 02:07:58.895725 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:58.895695 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e78415-5hzkc"] Jul 28 02:07:59.101340 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:07:59.101310 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40587140-7a6d-4151-bcbc-44f583ca6d36" path="/var/lib/kubelet/pods/40587140-7a6d-4151-bcbc-44f583ca6d36/volumes" Jul 28 02:08:09.282575 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:09.282489 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-49ecf4-j6q9f"] Jul 28 02:08:09.282949 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:09.282865 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40587140-7a6d-4151-bcbc-44f583ca6d36" containerName="revoke-keys" Jul 28 02:08:09.282949 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:09.282878 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="40587140-7a6d-4151-bcbc-44f583ca6d36" containerName="revoke-keys" Jul 28 02:08:09.282949 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:09.282888 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03fb507c-6657-4b97-b7d8-75ed80a61593" containerName="maas-api" Jul 28 02:08:09.282949 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:09.282895 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="03fb507c-6657-4b97-b7d8-75ed80a61593" containerName="maas-api" Jul 28 02:08:09.282949 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:09.282951 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="40587140-7a6d-4151-bcbc-44f583ca6d36" containerName="revoke-keys" Jul 28 02:08:09.283135 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:09.282959 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="03fb507c-6657-4b97-b7d8-75ed80a61593" containerName="maas-api" Jul 28 02:08:09.287329 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:09.287311 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-49ecf4-j6q9f" Jul 28 02:08:09.290094 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:09.290067 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wjxvt\"" Jul 28 02:08:09.292868 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:09.292841 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-49ecf4-j6q9f"] Jul 28 02:08:09.340952 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:09.340916 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb3f12c-e8c6-4780-bf14-d91404b2e9c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-49ecf4-j6q9f\" (UID: \"8cb3f12c-e8c6-4780-bf14-d91404b2e9c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-49ecf4-j6q9f" Jul 28 02:08:09.442444 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:09.442410 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb3f12c-e8c6-4780-bf14-d91404b2e9c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-49ecf4-j6q9f\" (UID: \"8cb3f12c-e8c6-4780-bf14-d91404b2e9c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-49ecf4-j6q9f" Jul 28 02:08:09.443083 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:09.443061 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb3f12c-e8c6-4780-bf14-d91404b2e9c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-49ecf4-j6q9f\" (UID: \"8cb3f12c-e8c6-4780-bf14-d91404b2e9c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-49ecf4-j6q9f" Jul 28 02:08:09.599939 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:09.599840 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-49ecf4-j6q9f" Jul 28 02:08:09.732802 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:09.732774 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-49ecf4-j6q9f"] Jul 28 02:08:09.734117 ip-10-0-131-23 kubenswrapper[2565]: W0728 02:08:09.734082 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cb3f12c_e8c6_4780_bf14_d91404b2e9c5.slice/crio-9ffd04dc4e544c89297bc88dac94b344df4db50796b9c16be861ee185d8c7ad9 WatchSource:0}: Error finding container 9ffd04dc4e544c89297bc88dac94b344df4db50796b9c16be861ee185d8c7ad9: Status 404 returned error can't find the container with id 9ffd04dc4e544c89297bc88dac94b344df4db50796b9c16be861ee185d8c7ad9 Jul 28 02:08:10.580364 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:10.580266 2565 generic.go:356] "Generic (PLEG): container finished" podID="8cb3f12c-e8c6-4780-bf14-d91404b2e9c5" containerID="60e681f20c99f522d14257eefcdf6ff3a64f7917138f894c14c795f209fb6279" exitCode=0 Jul 28 02:08:10.580364 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:10.580345 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-49ecf4-j6q9f" event={"ID":"8cb3f12c-e8c6-4780-bf14-d91404b2e9c5","Type":"ContainerDied","Data":"60e681f20c99f522d14257eefcdf6ff3a64f7917138f894c14c795f209fb6279"} Jul 28 02:08:10.580775 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:10.580398 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-49ecf4-j6q9f" event={"ID":"8cb3f12c-e8c6-4780-bf14-d91404b2e9c5","Type":"ContainerStarted","Data":"9ffd04dc4e544c89297bc88dac94b344df4db50796b9c16be861ee185d8c7ad9"} Jul 28 02:08:11.720183 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:11.720155 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-49ecf4-j6q9f" Jul 28 02:08:11.768318 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:11.768286 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb3f12c-e8c6-4780-bf14-d91404b2e9c5-maas-api-service-ca\") pod \"8cb3f12c-e8c6-4780-bf14-d91404b2e9c5\" (UID: \"8cb3f12c-e8c6-4780-bf14-d91404b2e9c5\") " Jul 28 02:08:11.768698 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:11.768677 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cb3f12c-e8c6-4780-bf14-d91404b2e9c5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8cb3f12c-e8c6-4780-bf14-d91404b2e9c5" (UID: "8cb3f12c-e8c6-4780-bf14-d91404b2e9c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 02:08:11.869758 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:11.869655 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb3f12c-e8c6-4780-bf14-d91404b2e9c5-maas-api-service-ca\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:08:12.592745 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:12.592714 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-49ecf4-j6q9f" Jul 28 02:08:12.592930 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:12.592724 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-49ecf4-j6q9f" event={"ID":"8cb3f12c-e8c6-4780-bf14-d91404b2e9c5","Type":"ContainerDied","Data":"9ffd04dc4e544c89297bc88dac94b344df4db50796b9c16be861ee185d8c7ad9"} Jul 28 02:08:12.592930 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:12.592835 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ffd04dc4e544c89297bc88dac94b344df4db50796b9c16be861ee185d8c7ad9" Jul 28 02:08:28.811559 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:28.811519 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-49ecf4-84d95869cb-vkddb"] Jul 28 02:08:28.812137 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:28.811717 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49ecf4-84d95869cb-vkddb" podUID="fc0ceaac-b443-4868-811e-60201f68c659" containerName="maas-api" containerID="cri-o://eb83e9d35d43e903643a8c41fcd1bf409c9b0ce17ae0fd5317b498231b772e78" gracePeriod=30 Jul 28 02:08:29.074495 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:29.074431 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49ecf4-84d95869cb-vkddb" Jul 28 02:08:29.245153 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:29.245122 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc0ceaac-b443-4868-811e-60201f68c659-maas-api-tls\") pod \"fc0ceaac-b443-4868-811e-60201f68c659\" (UID: \"fc0ceaac-b443-4868-811e-60201f68c659\") " Jul 28 02:08:29.245336 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:29.245172 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc0ceaac-b443-4868-811e-60201f68c659-openshift-service-ca\") pod \"fc0ceaac-b443-4868-811e-60201f68c659\" (UID: \"fc0ceaac-b443-4868-811e-60201f68c659\") " Jul 28 02:08:29.245336 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:29.245237 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9h4rc\" (UniqueName: \"kubernetes.io/projected/fc0ceaac-b443-4868-811e-60201f68c659-kube-api-access-9h4rc\") pod \"fc0ceaac-b443-4868-811e-60201f68c659\" (UID: \"fc0ceaac-b443-4868-811e-60201f68c659\") " Jul 28 02:08:29.245617 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:29.245583 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc0ceaac-b443-4868-811e-60201f68c659-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fc0ceaac-b443-4868-811e-60201f68c659" (UID: "fc0ceaac-b443-4868-811e-60201f68c659"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 02:08:29.245826 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:29.245801 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc0ceaac-b443-4868-811e-60201f68c659-openshift-service-ca\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:08:29.247291 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:29.247261 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc0ceaac-b443-4868-811e-60201f68c659-kube-api-access-9h4rc" (OuterVolumeSpecName: "kube-api-access-9h4rc") pod "fc0ceaac-b443-4868-811e-60201f68c659" (UID: "fc0ceaac-b443-4868-811e-60201f68c659"). InnerVolumeSpecName "kube-api-access-9h4rc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 02:08:29.247390 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:29.247314 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc0ceaac-b443-4868-811e-60201f68c659-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fc0ceaac-b443-4868-811e-60201f68c659" (UID: "fc0ceaac-b443-4868-811e-60201f68c659"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 02:08:29.346810 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:29.346727 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h4rc\" (UniqueName: \"kubernetes.io/projected/fc0ceaac-b443-4868-811e-60201f68c659-kube-api-access-9h4rc\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:08:29.346810 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:29.346758 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc0ceaac-b443-4868-811e-60201f68c659-maas-api-tls\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:08:29.395199 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:29.395166 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-49ecf4-j6q9f"] Jul 28 02:08:29.398940 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:29.398915 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-49ecf4-j6q9f"] Jul 28 02:08:29.679007 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:29.678906 2565 generic.go:356] "Generic (PLEG): container finished" podID="fc0ceaac-b443-4868-811e-60201f68c659" containerID="eb83e9d35d43e903643a8c41fcd1bf409c9b0ce17ae0fd5317b498231b772e78" exitCode=0 Jul 28 02:08:29.679007 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:29.678959 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49ecf4-84d95869cb-vkddb" Jul 28 02:08:29.679007 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:29.678979 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49ecf4-84d95869cb-vkddb" event={"ID":"fc0ceaac-b443-4868-811e-60201f68c659","Type":"ContainerDied","Data":"eb83e9d35d43e903643a8c41fcd1bf409c9b0ce17ae0fd5317b498231b772e78"} Jul 28 02:08:29.679333 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:29.679023 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-49ecf4-84d95869cb-vkddb" event={"ID":"fc0ceaac-b443-4868-811e-60201f68c659","Type":"ContainerDied","Data":"1e8904ef2cc39ea85a45582f48be3ee7fbdd1d4a347858757b18baa0fabf7ed3"} Jul 28 02:08:29.679333 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:29.679085 2565 scope.go:117] "RemoveContainer" containerID="eb83e9d35d43e903643a8c41fcd1bf409c9b0ce17ae0fd5317b498231b772e78" Jul 28 02:08:29.688276 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:29.688246 2565 scope.go:117] "RemoveContainer" containerID="eb83e9d35d43e903643a8c41fcd1bf409c9b0ce17ae0fd5317b498231b772e78" Jul 28 02:08:29.688534 ip-10-0-131-23 kubenswrapper[2565]: E0728 02:08:29.688514 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb83e9d35d43e903643a8c41fcd1bf409c9b0ce17ae0fd5317b498231b772e78\": container with ID starting with eb83e9d35d43e903643a8c41fcd1bf409c9b0ce17ae0fd5317b498231b772e78 not found: ID does not exist" containerID="eb83e9d35d43e903643a8c41fcd1bf409c9b0ce17ae0fd5317b498231b772e78" Jul 28 02:08:29.688591 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:29.688541 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb83e9d35d43e903643a8c41fcd1bf409c9b0ce17ae0fd5317b498231b772e78"} err="failed to get container status \"eb83e9d35d43e903643a8c41fcd1bf409c9b0ce17ae0fd5317b498231b772e78\": rpc error: code = NotFound desc = could not find container \"eb83e9d35d43e903643a8c41fcd1bf409c9b0ce17ae0fd5317b498231b772e78\": container with ID starting with eb83e9d35d43e903643a8c41fcd1bf409c9b0ce17ae0fd5317b498231b772e78 not found: ID does not exist" Jul 28 02:08:29.716278 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:29.716237 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-49ecf4-84d95869cb-vkddb"] Jul 28 02:08:29.749013 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:29.748977 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-49ecf4-84d95869cb-vkddb"] Jul 28 02:08:31.101568 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:31.101536 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cb3f12c-e8c6-4780-bf14-d91404b2e9c5" path="/var/lib/kubelet/pods/8cb3f12c-e8c6-4780-bf14-d91404b2e9c5/volumes" Jul 28 02:08:31.101962 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:31.101944 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc0ceaac-b443-4868-811e-60201f68c659" path="/var/lib/kubelet/pods/fc0ceaac-b443-4868-811e-60201f68c659/volumes" Jul 28 02:08:34.819143 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:34.819107 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv"] Jul 28 02:08:49.838162 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:49.838111 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv" podUID="a2b2286b-40ac-41b0-9aac-64ed107953e8" containerName="main" containerID="cri-o://f0c23497381a286ea7c77f2473a91469ad5c44c4b163c433325a13ae709b38e1" gracePeriod=45 Jul 28 02:08:50.097447 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:50.097379 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv" Jul 28 02:08:50.240483 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:50.240444 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2b2286b-40ac-41b0-9aac-64ed107953e8-tls-certs\") pod \"a2b2286b-40ac-41b0-9aac-64ed107953e8\" (UID: \"a2b2286b-40ac-41b0-9aac-64ed107953e8\") " Jul 28 02:08:50.240673 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:50.240520 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2b2286b-40ac-41b0-9aac-64ed107953e8-model-cache\") pod \"a2b2286b-40ac-41b0-9aac-64ed107953e8\" (UID: \"a2b2286b-40ac-41b0-9aac-64ed107953e8\") " Jul 28 02:08:50.240673 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:50.240548 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2b2286b-40ac-41b0-9aac-64ed107953e8-home\") pod \"a2b2286b-40ac-41b0-9aac-64ed107953e8\" (UID: \"a2b2286b-40ac-41b0-9aac-64ed107953e8\") " Jul 28 02:08:50.240673 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:50.240635 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kn8qt\" (UniqueName: \"kubernetes.io/projected/a2b2286b-40ac-41b0-9aac-64ed107953e8-kube-api-access-kn8qt\") pod \"a2b2286b-40ac-41b0-9aac-64ed107953e8\" (UID: \"a2b2286b-40ac-41b0-9aac-64ed107953e8\") " Jul 28 02:08:50.240673 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:50.240661 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2b2286b-40ac-41b0-9aac-64ed107953e8-tmp-dir\") pod \"a2b2286b-40ac-41b0-9aac-64ed107953e8\" (UID: \"a2b2286b-40ac-41b0-9aac-64ed107953e8\") " Jul 28 02:08:50.240840 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:50.240705 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2b2286b-40ac-41b0-9aac-64ed107953e8-dshm\") pod \"a2b2286b-40ac-41b0-9aac-64ed107953e8\" (UID: \"a2b2286b-40ac-41b0-9aac-64ed107953e8\") " Jul 28 02:08:50.240895 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:50.240838 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b2286b-40ac-41b0-9aac-64ed107953e8-model-cache" (OuterVolumeSpecName: "model-cache") pod "a2b2286b-40ac-41b0-9aac-64ed107953e8" (UID: "a2b2286b-40ac-41b0-9aac-64ed107953e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 02:08:50.240953 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:50.240908 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b2286b-40ac-41b0-9aac-64ed107953e8-home" (OuterVolumeSpecName: "home") pod "a2b2286b-40ac-41b0-9aac-64ed107953e8" (UID: "a2b2286b-40ac-41b0-9aac-64ed107953e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 02:08:50.241018 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:50.240944 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b2286b-40ac-41b0-9aac-64ed107953e8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a2b2286b-40ac-41b0-9aac-64ed107953e8" (UID: "a2b2286b-40ac-41b0-9aac-64ed107953e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 02:08:50.241122 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:50.241055 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2b2286b-40ac-41b0-9aac-64ed107953e8-model-cache\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:08:50.241122 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:50.241069 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2b2286b-40ac-41b0-9aac-64ed107953e8-home\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:08:50.241122 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:50.241079 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2b2286b-40ac-41b0-9aac-64ed107953e8-tmp-dir\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:08:50.242804 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:50.242782 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b2286b-40ac-41b0-9aac-64ed107953e8-dshm" (OuterVolumeSpecName: "dshm") pod "a2b2286b-40ac-41b0-9aac-64ed107953e8" (UID: "a2b2286b-40ac-41b0-9aac-64ed107953e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 02:08:50.242899 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:50.242824 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2b2286b-40ac-41b0-9aac-64ed107953e8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a2b2286b-40ac-41b0-9aac-64ed107953e8" (UID: "a2b2286b-40ac-41b0-9aac-64ed107953e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 02:08:50.242941 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:50.242907 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2b2286b-40ac-41b0-9aac-64ed107953e8-kube-api-access-kn8qt" (OuterVolumeSpecName: "kube-api-access-kn8qt") pod "a2b2286b-40ac-41b0-9aac-64ed107953e8" (UID: "a2b2286b-40ac-41b0-9aac-64ed107953e8"). InnerVolumeSpecName "kube-api-access-kn8qt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 02:08:50.342567 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:50.342524 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn8qt\" (UniqueName: \"kubernetes.io/projected/a2b2286b-40ac-41b0-9aac-64ed107953e8-kube-api-access-kn8qt\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:08:50.342567 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:50.342561 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2b2286b-40ac-41b0-9aac-64ed107953e8-dshm\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:08:50.342567 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:50.342572 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2b2286b-40ac-41b0-9aac-64ed107953e8-tls-certs\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:08:50.790624 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:50.790589 2565 generic.go:356] "Generic (PLEG): container finished" podID="a2b2286b-40ac-41b0-9aac-64ed107953e8" containerID="f0c23497381a286ea7c77f2473a91469ad5c44c4b163c433325a13ae709b38e1" exitCode=0 Jul 28 02:08:50.790842 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:50.790646 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv" Jul 28 02:08:50.790842 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:50.790658 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv" event={"ID":"a2b2286b-40ac-41b0-9aac-64ed107953e8","Type":"ContainerDied","Data":"f0c23497381a286ea7c77f2473a91469ad5c44c4b163c433325a13ae709b38e1"} Jul 28 02:08:50.790842 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:50.790693 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv" event={"ID":"a2b2286b-40ac-41b0-9aac-64ed107953e8","Type":"ContainerDied","Data":"a35bc628196937d1fbffbe0312d4a0aedbc66bfe80427a281adee66f9af25ca4"} Jul 28 02:08:50.790842 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:50.790709 2565 scope.go:117] "RemoveContainer" containerID="f0c23497381a286ea7c77f2473a91469ad5c44c4b163c433325a13ae709b38e1" Jul 28 02:08:50.799864 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:50.799841 2565 scope.go:117] "RemoveContainer" containerID="f0c23497381a286ea7c77f2473a91469ad5c44c4b163c433325a13ae709b38e1" Jul 28 02:08:50.800169 ip-10-0-131-23 kubenswrapper[2565]: E0728 02:08:50.800145 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0c23497381a286ea7c77f2473a91469ad5c44c4b163c433325a13ae709b38e1\": container with ID starting with f0c23497381a286ea7c77f2473a91469ad5c44c4b163c433325a13ae709b38e1 not found: ID does not exist" containerID="f0c23497381a286ea7c77f2473a91469ad5c44c4b163c433325a13ae709b38e1" Jul 28 02:08:50.800261 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:50.800177 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0c23497381a286ea7c77f2473a91469ad5c44c4b163c433325a13ae709b38e1"} err="failed to get container status \"f0c23497381a286ea7c77f2473a91469ad5c44c4b163c433325a13ae709b38e1\": rpc error: code = NotFound desc = could not find container \"f0c23497381a286ea7c77f2473a91469ad5c44c4b163c433325a13ae709b38e1\": container with ID starting with f0c23497381a286ea7c77f2473a91469ad5c44c4b163c433325a13ae709b38e1 not found: ID does not exist" Jul 28 02:08:50.812328 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:50.812301 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv"] Jul 28 02:08:50.815517 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:50.815497 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-49ecf4/ipp-route-49ecf4-kserve-85f5cf5d59-nmdwv"] Jul 28 02:08:51.101782 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:08:51.101699 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2b2286b-40ac-41b0-9aac-64ed107953e8" path="/var/lib/kubelet/pods/a2b2286b-40ac-41b0-9aac-64ed107953e8/volumes" Jul 28 02:09:18.608282 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:18.608242 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e0a963-9ss2c"] Jul 28 02:09:18.608701 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:18.608604 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc0ceaac-b443-4868-811e-60201f68c659" containerName="maas-api" Jul 28 02:09:18.608701 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:18.608614 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc0ceaac-b443-4868-811e-60201f68c659" containerName="maas-api" Jul 28 02:09:18.608701 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:18.608624 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2b2286b-40ac-41b0-9aac-64ed107953e8" containerName="main" Jul 28 02:09:18.608701 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:18.608630 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2b2286b-40ac-41b0-9aac-64ed107953e8" containerName="main" Jul 28 02:09:18.608701 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:18.608636 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cb3f12c-e8c6-4780-bf14-d91404b2e9c5" containerName="revoke-keys" Jul 28 02:09:18.608701 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:18.608641 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cb3f12c-e8c6-4780-bf14-d91404b2e9c5" containerName="revoke-keys" Jul 28 02:09:18.608701 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:18.608695 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2b2286b-40ac-41b0-9aac-64ed107953e8" containerName="main" Jul 28 02:09:18.608701 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:18.608705 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc0ceaac-b443-4868-811e-60201f68c659" containerName="maas-api" Jul 28 02:09:18.608962 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:18.608713 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cb3f12c-e8c6-4780-bf14-d91404b2e9c5" containerName="revoke-keys" Jul 28 02:09:18.611777 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:18.611757 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e0a963-9ss2c" Jul 28 02:09:18.614703 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:18.614679 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wjxvt\"" Jul 28 02:09:18.621804 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:18.621774 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e0a963-9ss2c"] Jul 28 02:09:18.697562 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:18.697521 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e031ec1-f456-46ab-a563-74c4deff611c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e0a963-9ss2c\" (UID: \"5e031ec1-f456-46ab-a563-74c4deff611c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e0a963-9ss2c" Jul 28 02:09:18.798248 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:18.798208 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e031ec1-f456-46ab-a563-74c4deff611c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e0a963-9ss2c\" (UID: \"5e031ec1-f456-46ab-a563-74c4deff611c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e0a963-9ss2c" Jul 28 02:09:18.798850 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:18.798828 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e031ec1-f456-46ab-a563-74c4deff611c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e0a963-9ss2c\" (UID: \"5e031ec1-f456-46ab-a563-74c4deff611c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e0a963-9ss2c" Jul 28 02:09:18.923188 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:18.923095 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e0a963-9ss2c" Jul 28 02:09:19.060396 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:19.060367 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e0a963-9ss2c"] Jul 28 02:09:19.062268 ip-10-0-131-23 kubenswrapper[2565]: W0728 02:09:19.062239 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e031ec1_f456_46ab_a563_74c4deff611c.slice/crio-0b5f3b64c71805a43ea9afb0b189134fe6b760e19d52640523fbc7d832190dd0 WatchSource:0}: Error finding container 0b5f3b64c71805a43ea9afb0b189134fe6b760e19d52640523fbc7d832190dd0: Status 404 returned error can't find the container with id 0b5f3b64c71805a43ea9afb0b189134fe6b760e19d52640523fbc7d832190dd0 Jul 28 02:09:19.944895 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:19.944862 2565 generic.go:356] "Generic (PLEG): container finished" podID="5e031ec1-f456-46ab-a563-74c4deff611c" containerID="2beba3c990d1990e5953fa97fce4ea3edb9b1fbc9d2efab154d81fd7bf22021b" exitCode=0 Jul 28 02:09:19.945304 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:19.944953 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e0a963-9ss2c" event={"ID":"5e031ec1-f456-46ab-a563-74c4deff611c","Type":"ContainerDied","Data":"2beba3c990d1990e5953fa97fce4ea3edb9b1fbc9d2efab154d81fd7bf22021b"} Jul 28 02:09:19.945304 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:19.945011 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e0a963-9ss2c" event={"ID":"5e031ec1-f456-46ab-a563-74c4deff611c","Type":"ContainerStarted","Data":"0b5f3b64c71805a43ea9afb0b189134fe6b760e19d52640523fbc7d832190dd0"} Jul 28 02:09:21.081550 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:21.081524 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e0a963-9ss2c" Jul 28 02:09:21.121093 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:21.121063 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e031ec1-f456-46ab-a563-74c4deff611c-maas-api-service-ca\") pod \"5e031ec1-f456-46ab-a563-74c4deff611c\" (UID: \"5e031ec1-f456-46ab-a563-74c4deff611c\") " Jul 28 02:09:21.121415 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:21.121391 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e031ec1-f456-46ab-a563-74c4deff611c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5e031ec1-f456-46ab-a563-74c4deff611c" (UID: "5e031ec1-f456-46ab-a563-74c4deff611c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 02:09:21.222534 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:21.222443 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e031ec1-f456-46ab-a563-74c4deff611c-maas-api-service-ca\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:09:21.957132 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:21.957099 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e0a963-9ss2c" Jul 28 02:09:21.957132 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:21.957121 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e0a963-9ss2c" event={"ID":"5e031ec1-f456-46ab-a563-74c4deff611c","Type":"ContainerDied","Data":"0b5f3b64c71805a43ea9afb0b189134fe6b760e19d52640523fbc7d832190dd0"} Jul 28 02:09:21.957352 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:21.957173 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b5f3b64c71805a43ea9afb0b189134fe6b760e19d52640523fbc7d832190dd0" Jul 28 02:09:29.253204 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:29.253167 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k"] Jul 28 02:09:29.253689 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:29.253399 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k" podUID="45abba4e-fffa-4e78-b46b-e3367d8c6816" containerName="maas-api" containerID="cri-o://dbdc7e1f3ab52223eba9da559a6bcd049e5d5d237e6586d79be3d119a572a33c" gracePeriod=30 Jul 28 02:09:29.513008 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:29.512941 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k" Jul 28 02:09:29.599256 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:29.599219 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45abba4e-fffa-4e78-b46b-e3367d8c6816-openshift-service-ca\") pod \"45abba4e-fffa-4e78-b46b-e3367d8c6816\" (UID: \"45abba4e-fffa-4e78-b46b-e3367d8c6816\") " Jul 28 02:09:29.599466 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:29.599285 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45abba4e-fffa-4e78-b46b-e3367d8c6816-maas-api-tls\") pod \"45abba4e-fffa-4e78-b46b-e3367d8c6816\" (UID: \"45abba4e-fffa-4e78-b46b-e3367d8c6816\") " Jul 28 02:09:29.599466 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:29.599393 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gktck\" (UniqueName: \"kubernetes.io/projected/45abba4e-fffa-4e78-b46b-e3367d8c6816-kube-api-access-gktck\") pod \"45abba4e-fffa-4e78-b46b-e3367d8c6816\" (UID: \"45abba4e-fffa-4e78-b46b-e3367d8c6816\") " Jul 28 02:09:29.599684 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:29.599653 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45abba4e-fffa-4e78-b46b-e3367d8c6816-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "45abba4e-fffa-4e78-b46b-e3367d8c6816" (UID: "45abba4e-fffa-4e78-b46b-e3367d8c6816"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 02:09:29.599820 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:29.599770 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45abba4e-fffa-4e78-b46b-e3367d8c6816-openshift-service-ca\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:09:29.601542 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:29.601520 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45abba4e-fffa-4e78-b46b-e3367d8c6816-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "45abba4e-fffa-4e78-b46b-e3367d8c6816" (UID: "45abba4e-fffa-4e78-b46b-e3367d8c6816"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 02:09:29.601657 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:29.601561 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45abba4e-fffa-4e78-b46b-e3367d8c6816-kube-api-access-gktck" (OuterVolumeSpecName: "kube-api-access-gktck") pod "45abba4e-fffa-4e78-b46b-e3367d8c6816" (UID: "45abba4e-fffa-4e78-b46b-e3367d8c6816"). InnerVolumeSpecName "kube-api-access-gktck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 02:09:29.701158 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:29.701127 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45abba4e-fffa-4e78-b46b-e3367d8c6816-maas-api-tls\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:09:29.701158 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:29.701153 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gktck\" (UniqueName: \"kubernetes.io/projected/45abba4e-fffa-4e78-b46b-e3367d8c6816-kube-api-access-gktck\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:09:29.999665 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:29.999630 2565 generic.go:356] "Generic (PLEG): container finished" podID="45abba4e-fffa-4e78-b46b-e3367d8c6816" containerID="dbdc7e1f3ab52223eba9da559a6bcd049e5d5d237e6586d79be3d119a572a33c" exitCode=0 Jul 28 02:09:29.999885 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:29.999687 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k" Jul 28 02:09:29.999885 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:29.999726 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k" event={"ID":"45abba4e-fffa-4e78-b46b-e3367d8c6816","Type":"ContainerDied","Data":"dbdc7e1f3ab52223eba9da559a6bcd049e5d5d237e6586d79be3d119a572a33c"} Jul 28 02:09:29.999885 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:29.999779 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k" event={"ID":"45abba4e-fffa-4e78-b46b-e3367d8c6816","Type":"ContainerDied","Data":"1966d3892ef13658d4afb7347c4aee7f537b99596fa90e56667ed85a79150226"} Jul 28 02:09:29.999885 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:29.999808 2565 scope.go:117] "RemoveContainer" containerID="dbdc7e1f3ab52223eba9da559a6bcd049e5d5d237e6586d79be3d119a572a33c" Jul 28 02:09:30.008920 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:30.008901 2565 scope.go:117] "RemoveContainer" containerID="dbdc7e1f3ab52223eba9da559a6bcd049e5d5d237e6586d79be3d119a572a33c" Jul 28 02:09:30.009199 ip-10-0-131-23 kubenswrapper[2565]: E0728 02:09:30.009182 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbdc7e1f3ab52223eba9da559a6bcd049e5d5d237e6586d79be3d119a572a33c\": container with ID starting with dbdc7e1f3ab52223eba9da559a6bcd049e5d5d237e6586d79be3d119a572a33c not found: ID does not exist" containerID="dbdc7e1f3ab52223eba9da559a6bcd049e5d5d237e6586d79be3d119a572a33c" Jul 28 02:09:30.009249 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:30.009207 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbdc7e1f3ab52223eba9da559a6bcd049e5d5d237e6586d79be3d119a572a33c"} err="failed to get container status \"dbdc7e1f3ab52223eba9da559a6bcd049e5d5d237e6586d79be3d119a572a33c\": rpc error: code = NotFound desc = could not find container \"dbdc7e1f3ab52223eba9da559a6bcd049e5d5d237e6586d79be3d119a572a33c\": container with ID starting with dbdc7e1f3ab52223eba9da559a6bcd049e5d5d237e6586d79be3d119a572a33c not found: ID does not exist" Jul 28 02:09:30.020530 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:30.020504 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k"] Jul 28 02:09:30.024250 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:30.024224 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a963-99d87bd8b-7kh8k"] Jul 28 02:09:31.101404 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:31.101358 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45abba4e-fffa-4e78-b46b-e3367d8c6816" path="/var/lib/kubelet/pods/45abba4e-fffa-4e78-b46b-e3367d8c6816/volumes" Jul 28 02:09:33.714522 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:33.714474 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e0a963-9ss2c"] Jul 28 02:09:33.721057 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:33.721016 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e0a963-9ss2c"] Jul 28 02:09:35.101479 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:35.101398 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e031ec1-f456-46ab-a563-74c4deff611c" path="/var/lib/kubelet/pods/5e031ec1-f456-46ab-a563-74c4deff611c/volumes" Jul 28 02:09:52.866656 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:52.866614 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-dfed9b-766788bf-md4fq"] Jul 28 02:09:52.867191 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:52.866863 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dfed9b-766788bf-md4fq" podUID="c1b0ca2e-075a-40e7-b328-34d54d43b6d8" containerName="maas-api" containerID="cri-o://0e4b6cef7f0262a4e21267db86ad1651821f77efa8a62276bc34d6ea72ef2d76" gracePeriod=30 Jul 28 02:09:53.114945 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:53.114914 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dfed9b-766788bf-md4fq" Jul 28 02:09:53.119344 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:53.119255 2565 generic.go:356] "Generic (PLEG): container finished" podID="c1b0ca2e-075a-40e7-b328-34d54d43b6d8" containerID="0e4b6cef7f0262a4e21267db86ad1651821f77efa8a62276bc34d6ea72ef2d76" exitCode=0 Jul 28 02:09:53.119344 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:53.119308 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dfed9b-766788bf-md4fq" Jul 28 02:09:53.119551 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:53.119339 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dfed9b-766788bf-md4fq" event={"ID":"c1b0ca2e-075a-40e7-b328-34d54d43b6d8","Type":"ContainerDied","Data":"0e4b6cef7f0262a4e21267db86ad1651821f77efa8a62276bc34d6ea72ef2d76"} Jul 28 02:09:53.119551 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:53.119400 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dfed9b-766788bf-md4fq" event={"ID":"c1b0ca2e-075a-40e7-b328-34d54d43b6d8","Type":"ContainerDied","Data":"6d12212538b2a7c44cc0f572da63f230575db002310016b2ad57eed11c3bbb9c"} Jul 28 02:09:53.119551 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:53.119421 2565 scope.go:117] "RemoveContainer" containerID="0e4b6cef7f0262a4e21267db86ad1651821f77efa8a62276bc34d6ea72ef2d76" Jul 28 02:09:53.130431 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:53.130401 2565 scope.go:117] "RemoveContainer" containerID="0e4b6cef7f0262a4e21267db86ad1651821f77efa8a62276bc34d6ea72ef2d76" Jul 28 02:09:53.130765 ip-10-0-131-23 kubenswrapper[2565]: E0728 02:09:53.130735 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e4b6cef7f0262a4e21267db86ad1651821f77efa8a62276bc34d6ea72ef2d76\": container with ID starting with 0e4b6cef7f0262a4e21267db86ad1651821f77efa8a62276bc34d6ea72ef2d76 not found: ID does not exist" containerID="0e4b6cef7f0262a4e21267db86ad1651821f77efa8a62276bc34d6ea72ef2d76" Jul 28 02:09:53.130845 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:53.130767 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e4b6cef7f0262a4e21267db86ad1651821f77efa8a62276bc34d6ea72ef2d76"} err="failed to get container status \"0e4b6cef7f0262a4e21267db86ad1651821f77efa8a62276bc34d6ea72ef2d76\": rpc error: code = NotFound desc = could not find container \"0e4b6cef7f0262a4e21267db86ad1651821f77efa8a62276bc34d6ea72ef2d76\": container with ID starting with 0e4b6cef7f0262a4e21267db86ad1651821f77efa8a62276bc34d6ea72ef2d76 not found: ID does not exist" Jul 28 02:09:53.231236 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:53.231204 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1b0ca2e-075a-40e7-b328-34d54d43b6d8-openshift-service-ca\") pod \"c1b0ca2e-075a-40e7-b328-34d54d43b6d8\" (UID: \"c1b0ca2e-075a-40e7-b328-34d54d43b6d8\") " Jul 28 02:09:53.231414 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:53.231249 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1b0ca2e-075a-40e7-b328-34d54d43b6d8-maas-api-tls\") pod \"c1b0ca2e-075a-40e7-b328-34d54d43b6d8\" (UID: \"c1b0ca2e-075a-40e7-b328-34d54d43b6d8\") " Jul 28 02:09:53.231414 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:53.231341 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82pjf\" (UniqueName: \"kubernetes.io/projected/c1b0ca2e-075a-40e7-b328-34d54d43b6d8-kube-api-access-82pjf\") pod \"c1b0ca2e-075a-40e7-b328-34d54d43b6d8\" (UID: \"c1b0ca2e-075a-40e7-b328-34d54d43b6d8\") " Jul 28 02:09:53.231657 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:53.231620 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1b0ca2e-075a-40e7-b328-34d54d43b6d8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c1b0ca2e-075a-40e7-b328-34d54d43b6d8" (UID: "c1b0ca2e-075a-40e7-b328-34d54d43b6d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 02:09:53.233811 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:53.233784 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b0ca2e-075a-40e7-b328-34d54d43b6d8-kube-api-access-82pjf" (OuterVolumeSpecName: "kube-api-access-82pjf") pod "c1b0ca2e-075a-40e7-b328-34d54d43b6d8" (UID: "c1b0ca2e-075a-40e7-b328-34d54d43b6d8"). InnerVolumeSpecName "kube-api-access-82pjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 02:09:53.233907 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:53.233823 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1b0ca2e-075a-40e7-b328-34d54d43b6d8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c1b0ca2e-075a-40e7-b328-34d54d43b6d8" (UID: "c1b0ca2e-075a-40e7-b328-34d54d43b6d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 02:09:53.332380 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:53.332340 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1b0ca2e-075a-40e7-b328-34d54d43b6d8-openshift-service-ca\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:09:53.332380 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:53.332374 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1b0ca2e-075a-40e7-b328-34d54d43b6d8-maas-api-tls\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:09:53.332380 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:53.332383 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82pjf\" (UniqueName: \"kubernetes.io/projected/c1b0ca2e-075a-40e7-b328-34d54d43b6d8-kube-api-access-82pjf\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:09:53.441536 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:53.441499 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-dfed9b-766788bf-md4fq"] Jul 28 02:09:53.446679 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:53.446652 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-dfed9b-766788bf-md4fq"] Jul 28 02:09:55.101708 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:09:55.101668 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1b0ca2e-075a-40e7-b328-34d54d43b6d8" path="/var/lib/kubelet/pods/c1b0ca2e-075a-40e7-b328-34d54d43b6d8/volumes" Jul 28 02:10:13.812761 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:13.812720 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-de398f-558b47988c-s7nvq"] Jul 28 02:10:13.813246 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:13.812959 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de398f-558b47988c-s7nvq" podUID="95c353c3-777d-4f8e-9eea-3055b77db4d6" containerName="maas-api" containerID="cri-o://eb5e1da5aa565ef23ba23d3ed4392e79e5bb6c87ca3f4f8f272568444aad53d1" gracePeriod=30 Jul 28 02:10:13.979406 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:13.979358 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de398f-558b47988c-s7nvq" podUID="95c353c3-777d-4f8e-9eea-3055b77db4d6" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.58:8443/health\": dial tcp 10.133.0.58:8443: connect: connection refused" Jul 28 02:10:14.072125 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:14.072020 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de398f-558b47988c-s7nvq" Jul 28 02:10:14.105275 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:14.105247 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c353c3-777d-4f8e-9eea-3055b77db4d6-openshift-service-ca\") pod \"95c353c3-777d-4f8e-9eea-3055b77db4d6\" (UID: \"95c353c3-777d-4f8e-9eea-3055b77db4d6\") " Jul 28 02:10:14.105468 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:14.105303 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6228\" (UniqueName: \"kubernetes.io/projected/95c353c3-777d-4f8e-9eea-3055b77db4d6-kube-api-access-f6228\") pod \"95c353c3-777d-4f8e-9eea-3055b77db4d6\" (UID: \"95c353c3-777d-4f8e-9eea-3055b77db4d6\") " Jul 28 02:10:14.105468 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:14.105351 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95c353c3-777d-4f8e-9eea-3055b77db4d6-maas-api-tls\") pod \"95c353c3-777d-4f8e-9eea-3055b77db4d6\" (UID: \"95c353c3-777d-4f8e-9eea-3055b77db4d6\") " Jul 28 02:10:14.105670 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:14.105645 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95c353c3-777d-4f8e-9eea-3055b77db4d6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "95c353c3-777d-4f8e-9eea-3055b77db4d6" (UID: "95c353c3-777d-4f8e-9eea-3055b77db4d6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 02:10:14.107453 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:14.107419 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c353c3-777d-4f8e-9eea-3055b77db4d6-kube-api-access-f6228" (OuterVolumeSpecName: "kube-api-access-f6228") pod "95c353c3-777d-4f8e-9eea-3055b77db4d6" (UID: "95c353c3-777d-4f8e-9eea-3055b77db4d6"). InnerVolumeSpecName "kube-api-access-f6228". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 02:10:14.107453 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:14.107420 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95c353c3-777d-4f8e-9eea-3055b77db4d6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "95c353c3-777d-4f8e-9eea-3055b77db4d6" (UID: "95c353c3-777d-4f8e-9eea-3055b77db4d6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 02:10:14.206980 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:14.206936 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c353c3-777d-4f8e-9eea-3055b77db4d6-openshift-service-ca\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:10:14.206980 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:14.206975 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6228\" (UniqueName: \"kubernetes.io/projected/95c353c3-777d-4f8e-9eea-3055b77db4d6-kube-api-access-f6228\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:10:14.207297 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:14.206991 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95c353c3-777d-4f8e-9eea-3055b77db4d6-maas-api-tls\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:10:14.226532 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:14.226498 2565 generic.go:356] "Generic (PLEG): container finished" podID="95c353c3-777d-4f8e-9eea-3055b77db4d6" containerID="eb5e1da5aa565ef23ba23d3ed4392e79e5bb6c87ca3f4f8f272568444aad53d1" exitCode=0 Jul 28 02:10:14.226671 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:14.226536 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de398f-558b47988c-s7nvq" Jul 28 02:10:14.226671 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:14.226586 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de398f-558b47988c-s7nvq" event={"ID":"95c353c3-777d-4f8e-9eea-3055b77db4d6","Type":"ContainerDied","Data":"eb5e1da5aa565ef23ba23d3ed4392e79e5bb6c87ca3f4f8f272568444aad53d1"} Jul 28 02:10:14.226671 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:14.226631 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de398f-558b47988c-s7nvq" event={"ID":"95c353c3-777d-4f8e-9eea-3055b77db4d6","Type":"ContainerDied","Data":"9682ea5c5379b7701dabe5195fbdf194c0a365f88acb6326332a882cc4284efc"} Jul 28 02:10:14.226671 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:14.226654 2565 scope.go:117] "RemoveContainer" containerID="eb5e1da5aa565ef23ba23d3ed4392e79e5bb6c87ca3f4f8f272568444aad53d1" Jul 28 02:10:14.236400 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:14.236382 2565 scope.go:117] "RemoveContainer" containerID="eb5e1da5aa565ef23ba23d3ed4392e79e5bb6c87ca3f4f8f272568444aad53d1" Jul 28 02:10:14.236658 ip-10-0-131-23 kubenswrapper[2565]: E0728 02:10:14.236641 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb5e1da5aa565ef23ba23d3ed4392e79e5bb6c87ca3f4f8f272568444aad53d1\": container with ID starting with eb5e1da5aa565ef23ba23d3ed4392e79e5bb6c87ca3f4f8f272568444aad53d1 not found: ID does not exist" containerID="eb5e1da5aa565ef23ba23d3ed4392e79e5bb6c87ca3f4f8f272568444aad53d1" Jul 28 02:10:14.236716 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:14.236669 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb5e1da5aa565ef23ba23d3ed4392e79e5bb6c87ca3f4f8f272568444aad53d1"} err="failed to get container status \"eb5e1da5aa565ef23ba23d3ed4392e79e5bb6c87ca3f4f8f272568444aad53d1\": rpc error: code = NotFound desc = could not find container \"eb5e1da5aa565ef23ba23d3ed4392e79e5bb6c87ca3f4f8f272568444aad53d1\": container with ID starting with eb5e1da5aa565ef23ba23d3ed4392e79e5bb6c87ca3f4f8f272568444aad53d1 not found: ID does not exist" Jul 28 02:10:14.248562 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:14.248539 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-de398f-558b47988c-s7nvq"] Jul 28 02:10:14.254726 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:14.254703 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-de398f-558b47988c-s7nvq"] Jul 28 02:10:15.100988 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:15.100952 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95c353c3-777d-4f8e-9eea-3055b77db4d6" path="/var/lib/kubelet/pods/95c353c3-777d-4f8e-9eea-3055b77db4d6/volumes" Jul 28 02:10:25.347434 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:25.347400 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6"] Jul 28 02:10:25.348099 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:25.347634 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" podUID="8f99f783-4ef6-45b4-9692-02b8103b8f53" containerName="istio-proxy" containerID="cri-o://efc1163baa24546df5efbbc1183c548232bfbabdd27f2e366284bba76938f34e" gracePeriod=30 Jul 28 02:10:30.607504 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:30.607474 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 02:10:30.753570 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:30.753536 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f99f783-4ef6-45b4-9692-02b8103b8f53-credential-socket\") pod \"8f99f783-4ef6-45b4-9692-02b8103b8f53\" (UID: \"8f99f783-4ef6-45b4-9692-02b8103b8f53\") " Jul 28 02:10:30.753570 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:30.753578 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f99f783-4ef6-45b4-9692-02b8103b8f53-workload-socket\") pod \"8f99f783-4ef6-45b4-9692-02b8103b8f53\" (UID: \"8f99f783-4ef6-45b4-9692-02b8103b8f53\") " Jul 28 02:10:30.753835 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:30.753624 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f99f783-4ef6-45b4-9692-02b8103b8f53-workload-certs\") pod \"8f99f783-4ef6-45b4-9692-02b8103b8f53\" (UID: \"8f99f783-4ef6-45b4-9692-02b8103b8f53\") " Jul 28 02:10:30.753835 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:30.753645 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f99f783-4ef6-45b4-9692-02b8103b8f53-istio-envoy\") pod \"8f99f783-4ef6-45b4-9692-02b8103b8f53\" (UID: \"8f99f783-4ef6-45b4-9692-02b8103b8f53\") " Jul 28 02:10:30.753835 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:30.753685 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f99f783-4ef6-45b4-9692-02b8103b8f53-istio-podinfo\") pod \"8f99f783-4ef6-45b4-9692-02b8103b8f53\" (UID: \"8f99f783-4ef6-45b4-9692-02b8103b8f53\") " Jul 28 02:10:30.753835 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:30.753755 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f99f783-4ef6-45b4-9692-02b8103b8f53-istio-token\") pod \"8f99f783-4ef6-45b4-9692-02b8103b8f53\" (UID: \"8f99f783-4ef6-45b4-9692-02b8103b8f53\") " Jul 28 02:10:30.753835 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:30.753794 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f99f783-4ef6-45b4-9692-02b8103b8f53-istiod-ca-cert\") pod \"8f99f783-4ef6-45b4-9692-02b8103b8f53\" (UID: \"8f99f783-4ef6-45b4-9692-02b8103b8f53\") " Jul 28 02:10:30.753835 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:30.753821 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f99f783-4ef6-45b4-9692-02b8103b8f53-istio-data\") pod \"8f99f783-4ef6-45b4-9692-02b8103b8f53\" (UID: \"8f99f783-4ef6-45b4-9692-02b8103b8f53\") " Jul 28 02:10:30.754180 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:30.753847 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zq9db\" (UniqueName: \"kubernetes.io/projected/8f99f783-4ef6-45b4-9692-02b8103b8f53-kube-api-access-zq9db\") pod \"8f99f783-4ef6-45b4-9692-02b8103b8f53\" (UID: \"8f99f783-4ef6-45b4-9692-02b8103b8f53\") " Jul 28 02:10:30.754180 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:30.753875 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f99f783-4ef6-45b4-9692-02b8103b8f53-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8f99f783-4ef6-45b4-9692-02b8103b8f53" (UID: "8f99f783-4ef6-45b4-9692-02b8103b8f53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 02:10:30.754180 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:30.753931 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f99f783-4ef6-45b4-9692-02b8103b8f53-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8f99f783-4ef6-45b4-9692-02b8103b8f53" (UID: "8f99f783-4ef6-45b4-9692-02b8103b8f53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 02:10:30.754180 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:30.754150 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f99f783-4ef6-45b4-9692-02b8103b8f53-credential-socket\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:10:30.754180 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:30.754168 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f99f783-4ef6-45b4-9692-02b8103b8f53-workload-socket\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:10:30.754448 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:30.754356 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f99f783-4ef6-45b4-9692-02b8103b8f53-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8f99f783-4ef6-45b4-9692-02b8103b8f53" (UID: "8f99f783-4ef6-45b4-9692-02b8103b8f53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 02:10:30.754448 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:30.754424 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f99f783-4ef6-45b4-9692-02b8103b8f53-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8f99f783-4ef6-45b4-9692-02b8103b8f53" (UID: "8f99f783-4ef6-45b4-9692-02b8103b8f53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 02:10:30.754756 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:30.754730 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f99f783-4ef6-45b4-9692-02b8103b8f53-istio-data" (OuterVolumeSpecName: "istio-data") pod "8f99f783-4ef6-45b4-9692-02b8103b8f53" (UID: "8f99f783-4ef6-45b4-9692-02b8103b8f53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 02:10:30.756177 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:30.756131 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f99f783-4ef6-45b4-9692-02b8103b8f53-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8f99f783-4ef6-45b4-9692-02b8103b8f53" (UID: "8f99f783-4ef6-45b4-9692-02b8103b8f53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 02:10:30.756307 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:30.756186 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f99f783-4ef6-45b4-9692-02b8103b8f53-kube-api-access-zq9db" (OuterVolumeSpecName: "kube-api-access-zq9db") pod "8f99f783-4ef6-45b4-9692-02b8103b8f53" (UID: "8f99f783-4ef6-45b4-9692-02b8103b8f53"). InnerVolumeSpecName "kube-api-access-zq9db". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 02:10:30.756307 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:30.756183 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f99f783-4ef6-45b4-9692-02b8103b8f53-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8f99f783-4ef6-45b4-9692-02b8103b8f53" (UID: "8f99f783-4ef6-45b4-9692-02b8103b8f53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 02:10:30.756307 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:30.756215 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f99f783-4ef6-45b4-9692-02b8103b8f53-istio-token" (OuterVolumeSpecName: "istio-token") pod "8f99f783-4ef6-45b4-9692-02b8103b8f53" (UID: "8f99f783-4ef6-45b4-9692-02b8103b8f53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 02:10:30.855046 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:30.855005 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq9db\" (UniqueName: \"kubernetes.io/projected/8f99f783-4ef6-45b4-9692-02b8103b8f53-kube-api-access-zq9db\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:10:30.855239 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:30.855065 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f99f783-4ef6-45b4-9692-02b8103b8f53-workload-certs\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:10:30.855239 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:30.855077 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f99f783-4ef6-45b4-9692-02b8103b8f53-istio-envoy\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:10:30.855239 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:30.855089 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f99f783-4ef6-45b4-9692-02b8103b8f53-istio-podinfo\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:10:30.855239 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:30.855100 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f99f783-4ef6-45b4-9692-02b8103b8f53-istio-token\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:10:30.855239 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:30.855112 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f99f783-4ef6-45b4-9692-02b8103b8f53-istiod-ca-cert\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:10:30.855239 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:30.855122 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f99f783-4ef6-45b4-9692-02b8103b8f53-istio-data\") on node \"ip-10-0-131-23.ec2.internal\" DevicePath \"\"" Jul 28 02:10:31.313226 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:31.313194 2565 generic.go:356] "Generic (PLEG): container finished" podID="8f99f783-4ef6-45b4-9692-02b8103b8f53" containerID="efc1163baa24546df5efbbc1183c548232bfbabdd27f2e366284bba76938f34e" exitCode=0 Jul 28 02:10:31.313393 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:31.313244 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" Jul 28 02:10:31.313393 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:31.313289 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" event={"ID":"8f99f783-4ef6-45b4-9692-02b8103b8f53","Type":"ContainerDied","Data":"efc1163baa24546df5efbbc1183c548232bfbabdd27f2e366284bba76938f34e"} Jul 28 02:10:31.313393 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:31.313342 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6" event={"ID":"8f99f783-4ef6-45b4-9692-02b8103b8f53","Type":"ContainerDied","Data":"2e76524d7d0e6f8115e161f24f76d6aeeb2c29a66a6a82aaee20203452ec01b3"} Jul 28 02:10:31.313393 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:31.313364 2565 scope.go:117] "RemoveContainer" containerID="efc1163baa24546df5efbbc1183c548232bfbabdd27f2e366284bba76938f34e" Jul 28 02:10:31.322368 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:31.322341 2565 scope.go:117] "RemoveContainer" containerID="31cae47582fa81a9643776867335507cd32ac50c6e7edf9f7a28e1def90b6908" Jul 28 02:10:31.331113 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:31.331093 2565 scope.go:117] "RemoveContainer" containerID="efc1163baa24546df5efbbc1183c548232bfbabdd27f2e366284bba76938f34e" Jul 28 02:10:31.331404 ip-10-0-131-23 kubenswrapper[2565]: E0728 02:10:31.331383 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efc1163baa24546df5efbbc1183c548232bfbabdd27f2e366284bba76938f34e\": container with ID starting with efc1163baa24546df5efbbc1183c548232bfbabdd27f2e366284bba76938f34e not found: ID does not exist" containerID="efc1163baa24546df5efbbc1183c548232bfbabdd27f2e366284bba76938f34e" Jul 28 02:10:31.331467 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:31.331412 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc1163baa24546df5efbbc1183c548232bfbabdd27f2e366284bba76938f34e"} err="failed to get container status \"efc1163baa24546df5efbbc1183c548232bfbabdd27f2e366284bba76938f34e\": rpc error: code = NotFound desc = could not find container \"efc1163baa24546df5efbbc1183c548232bfbabdd27f2e366284bba76938f34e\": container with ID starting with efc1163baa24546df5efbbc1183c548232bfbabdd27f2e366284bba76938f34e not found: ID does not exist" Jul 28 02:10:31.331467 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:31.331431 2565 scope.go:117] "RemoveContainer" containerID="31cae47582fa81a9643776867335507cd32ac50c6e7edf9f7a28e1def90b6908" Jul 28 02:10:31.331683 ip-10-0-131-23 kubenswrapper[2565]: E0728 02:10:31.331663 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31cae47582fa81a9643776867335507cd32ac50c6e7edf9f7a28e1def90b6908\": container with ID starting with 31cae47582fa81a9643776867335507cd32ac50c6e7edf9f7a28e1def90b6908 not found: ID does not exist" containerID="31cae47582fa81a9643776867335507cd32ac50c6e7edf9f7a28e1def90b6908" Jul 28 02:10:31.331727 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:31.331690 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31cae47582fa81a9643776867335507cd32ac50c6e7edf9f7a28e1def90b6908"} err="failed to get container status \"31cae47582fa81a9643776867335507cd32ac50c6e7edf9f7a28e1def90b6908\": rpc error: code = NotFound desc = could not find container \"31cae47582fa81a9643776867335507cd32ac50c6e7edf9f7a28e1def90b6908\": container with ID starting with 31cae47582fa81a9643776867335507cd32ac50c6e7edf9f7a28e1def90b6908 not found: ID does not exist" Jul 28 02:10:31.332567 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:31.332548 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6"] Jul 28 02:10:31.337962 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:31.337940 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-de398f-openshift-default-5db7d9f57b-7bdc6"] Jul 28 02:10:31.847521 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:31.847492 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8699bb9d7c-6p952_867b0a1e-9249-4fbc-be63-e0495f3aff8b/copy-manifests/0.log" Jul 28 02:10:31.865628 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:31.865600 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8699bb9d7c-6p952_867b0a1e-9249-4fbc-be63-e0495f3aff8b/manager/0.log" Jul 28 02:10:32.666241 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:32.666209 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753385-kzj5q_869473ef-831f-406d-a325-0959e20d25c2/cleanup/0.log" Jul 28 02:10:33.106239 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:33.106207 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f99f783-4ef6-45b4-9692-02b8103b8f53" path="/var/lib/kubelet/pods/8f99f783-4ef6-45b4-9692-02b8103b8f53/volumes" Jul 28 02:10:33.640665 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:33.640629 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g7xcs_2ce1d280-4f01-4cae-8d9b-57d48994c94b/util/0.log" Jul 28 02:10:33.646651 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:33.646612 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g7xcs_2ce1d280-4f01-4cae-8d9b-57d48994c94b/pull/0.log" Jul 28 02:10:33.652633 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:33.652613 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g7xcs_2ce1d280-4f01-4cae-8d9b-57d48994c94b/extract/0.log" Jul 28 02:10:33.764259 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:33.764224 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07dpr9_fbfce4de-622d-48f4-af1e-eef982d353e2/util/0.log" Jul 28 02:10:33.773787 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:33.773759 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07dpr9_fbfce4de-622d-48f4-af1e-eef982d353e2/pull/0.log" Jul 28 02:10:33.784475 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:33.784448 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07dpr9_fbfce4de-622d-48f4-af1e-eef982d353e2/extract/0.log" Jul 28 02:10:33.902750 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:33.902658 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tzrmz_53045446-768f-47e4-9ef4-5eda1c261514/util/0.log" Jul 28 02:10:33.908986 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:33.908957 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tzrmz_53045446-768f-47e4-9ef4-5eda1c261514/pull/0.log" Jul 28 02:10:33.915380 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:33.915359 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tzrmz_53045446-768f-47e4-9ef4-5eda1c261514/extract/0.log" Jul 28 02:10:34.025015 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:34.024985 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tptfg_bb1f9dce-dee3-446c-a9bf-16764b20a710/util/0.log" Jul 28 02:10:34.031102 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:34.031079 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tptfg_bb1f9dce-dee3-446c-a9bf-16764b20a710/pull/0.log" Jul 28 02:10:34.036897 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:34.036874 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tptfg_bb1f9dce-dee3-446c-a9bf-16764b20a710/extract/0.log" Jul 28 02:10:35.021943 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:35.021899 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-p47xv_e779e369-b9ef-4a41-8596-ee74986e44f1/manager/0.log" Jul 28 02:10:35.366488 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:35.366398 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj_d56a28fd-a428-4306-95a9-065e8e92e8bc/istio-proxy/0.log" Jul 28 02:10:35.485308 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:35.485275 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-85hcz_15fcf533-e2cd-4e92-8891-1d27b46f6ea9/discovery/0.log" Jul 28 02:10:36.493905 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:36.493871 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-lv7wk_7dbcb09e-a463-427d-8e2a-0023c811ec1d/main/0.log" Jul 28 02:10:36.938176 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:36.938147 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-vswsx_7b9e79a0-d79e-46b4-8d5d-d28bd99d1e15/main/0.log" Jul 28 02:10:44.924174 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:44.924142 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8sv4q_12e2c22a-bb31-4f97-869a-3c89eb64f837/global-pull-secret-syncer/0.log" Jul 28 02:10:45.103162 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:45.103132 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xtl8m_d018a93b-4db3-45cb-8e11-919a45a35718/konnectivity-agent/0.log" Jul 28 02:10:45.154565 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:45.154531 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-23.ec2.internal_3ff7e6d6dc372649b853c14fc595bb8f/haproxy/0.log" Jul 28 02:10:49.591188 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:49.591145 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g7xcs_2ce1d280-4f01-4cae-8d9b-57d48994c94b/extract/0.log" Jul 28 02:10:49.617554 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:49.617524 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g7xcs_2ce1d280-4f01-4cae-8d9b-57d48994c94b/util/0.log" Jul 28 02:10:49.637196 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:49.637155 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g7xcs_2ce1d280-4f01-4cae-8d9b-57d48994c94b/pull/0.log" Jul 28 02:10:49.661686 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:49.661650 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07dpr9_fbfce4de-622d-48f4-af1e-eef982d353e2/extract/0.log" Jul 28 02:10:49.681007 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:49.680977 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07dpr9_fbfce4de-622d-48f4-af1e-eef982d353e2/util/0.log" Jul 28 02:10:49.703809 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:49.703781 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07dpr9_fbfce4de-622d-48f4-af1e-eef982d353e2/pull/0.log" Jul 28 02:10:49.726778 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:49.726736 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tzrmz_53045446-768f-47e4-9ef4-5eda1c261514/extract/0.log" Jul 28 02:10:49.745495 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:49.745471 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tzrmz_53045446-768f-47e4-9ef4-5eda1c261514/util/0.log" Jul 28 02:10:49.764679 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:49.764647 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tzrmz_53045446-768f-47e4-9ef4-5eda1c261514/pull/0.log" Jul 28 02:10:49.793280 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:49.793243 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tptfg_bb1f9dce-dee3-446c-a9bf-16764b20a710/extract/0.log" Jul 28 02:10:49.814575 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:49.814548 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tptfg_bb1f9dce-dee3-446c-a9bf-16764b20a710/util/0.log" Jul 28 02:10:49.840157 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:49.840114 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tptfg_bb1f9dce-dee3-446c-a9bf-16764b20a710/pull/0.log" Jul 28 02:10:50.624686 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:50.624648 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-p47xv_e779e369-b9ef-4a41-8596-ee74986e44f1/manager/0.log" Jul 28 02:10:52.156473 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:52.156432 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_57766b8d-735c-444b-8fde-5ac7e36a037d/alertmanager/0.log" Jul 28 02:10:52.182230 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:52.182194 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_57766b8d-735c-444b-8fde-5ac7e36a037d/config-reloader/0.log" Jul 28 02:10:52.198963 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:52.198822 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_57766b8d-735c-444b-8fde-5ac7e36a037d/kube-rbac-proxy-web/0.log" Jul 28 02:10:52.220301 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:52.220276 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_57766b8d-735c-444b-8fde-5ac7e36a037d/kube-rbac-proxy/0.log" Jul 28 02:10:52.241409 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:52.241370 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_57766b8d-735c-444b-8fde-5ac7e36a037d/kube-rbac-proxy-metric/0.log" Jul 28 02:10:52.265921 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:52.265894 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_57766b8d-735c-444b-8fde-5ac7e36a037d/prom-label-proxy/0.log" Jul 28 02:10:52.289435 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:52.289406 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_57766b8d-735c-444b-8fde-5ac7e36a037d/init-config-reloader/0.log" Jul 28 02:10:52.462295 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:52.462262 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6465894bc4-lxqgl_51fdc520-b7b9-45fc-b93f-c220cab87fa9/metrics-server/0.log" Jul 28 02:10:52.713611 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:52.713523 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qkgnd_94413823-494b-477d-9cc9-51f4bcb2e16f/node-exporter/0.log" Jul 28 02:10:52.739971 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:52.739939 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qkgnd_94413823-494b-477d-9cc9-51f4bcb2e16f/kube-rbac-proxy/0.log" Jul 28 02:10:52.763158 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:52.763134 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qkgnd_94413823-494b-477d-9cc9-51f4bcb2e16f/init-textfile/0.log" Jul 28 02:10:54.966561 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:54.966520 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wvjdf/perf-node-gather-daemonset-6jr69"] Jul 28 02:10:54.967094 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:54.966855 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95c353c3-777d-4f8e-9eea-3055b77db4d6" containerName="maas-api" Jul 28 02:10:54.967094 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:54.966867 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="95c353c3-777d-4f8e-9eea-3055b77db4d6" containerName="maas-api" Jul 28 02:10:54.967094 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:54.966877 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45abba4e-fffa-4e78-b46b-e3367d8c6816" containerName="maas-api" Jul 28 02:10:54.967094 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:54.966883 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="45abba4e-fffa-4e78-b46b-e3367d8c6816" containerName="maas-api" Jul 28 02:10:54.967094 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:54.966898 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f99f783-4ef6-45b4-9692-02b8103b8f53" containerName="istio-proxy" Jul 28 02:10:54.967094 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:54.966905 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f99f783-4ef6-45b4-9692-02b8103b8f53" containerName="istio-proxy" Jul 28 02:10:54.967094 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:54.966915 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1b0ca2e-075a-40e7-b328-34d54d43b6d8" containerName="maas-api" Jul 28 02:10:54.967094 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:54.966920 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1b0ca2e-075a-40e7-b328-34d54d43b6d8" containerName="maas-api" Jul 28 02:10:54.967094 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:54.966927 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e031ec1-f456-46ab-a563-74c4deff611c" containerName="revoke-keys" Jul 28 02:10:54.967094 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:54.966932 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e031ec1-f456-46ab-a563-74c4deff611c" containerName="revoke-keys" Jul 28 02:10:54.967094 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:54.966945 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f99f783-4ef6-45b4-9692-02b8103b8f53" containerName="istio-proxy" Jul 28 02:10:54.967094 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:54.966951 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f99f783-4ef6-45b4-9692-02b8103b8f53" containerName="istio-proxy" Jul 28 02:10:54.967094 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:54.967002 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e031ec1-f456-46ab-a563-74c4deff611c" containerName="revoke-keys" Jul 28 02:10:54.967094 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:54.967010 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f99f783-4ef6-45b4-9692-02b8103b8f53" containerName="istio-proxy" Jul 28 02:10:54.967094 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:54.967018 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1b0ca2e-075a-40e7-b328-34d54d43b6d8" containerName="maas-api" Jul 28 02:10:54.967094 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:54.967024 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="45abba4e-fffa-4e78-b46b-e3367d8c6816" containerName="maas-api" Jul 28 02:10:54.967094 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:54.967043 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="95c353c3-777d-4f8e-9eea-3055b77db4d6" containerName="maas-api" Jul 28 02:10:54.971789 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:54.971767 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wvjdf/perf-node-gather-daemonset-6jr69" Jul 28 02:10:54.972185 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:54.972158 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e1ddcbd2-35c4-4177-88bd-4086374dc0e3-lib-modules\") pod \"perf-node-gather-daemonset-6jr69\" (UID: \"e1ddcbd2-35c4-4177-88bd-4086374dc0e3\") " pod="openshift-must-gather-wvjdf/perf-node-gather-daemonset-6jr69" Jul 28 02:10:54.972317 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:54.972248 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e1ddcbd2-35c4-4177-88bd-4086374dc0e3-proc\") pod \"perf-node-gather-daemonset-6jr69\" (UID: \"e1ddcbd2-35c4-4177-88bd-4086374dc0e3\") " pod="openshift-must-gather-wvjdf/perf-node-gather-daemonset-6jr69" Jul 28 02:10:54.972317 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:54.972292 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1ddcbd2-35c4-4177-88bd-4086374dc0e3-sys\") pod \"perf-node-gather-daemonset-6jr69\" (UID: \"e1ddcbd2-35c4-4177-88bd-4086374dc0e3\") " pod="openshift-must-gather-wvjdf/perf-node-gather-daemonset-6jr69" Jul 28 02:10:54.972437 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:54.972313 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sf27\" (UniqueName: \"kubernetes.io/projected/e1ddcbd2-35c4-4177-88bd-4086374dc0e3-kube-api-access-7sf27\") pod \"perf-node-gather-daemonset-6jr69\" (UID: \"e1ddcbd2-35c4-4177-88bd-4086374dc0e3\") " pod="openshift-must-gather-wvjdf/perf-node-gather-daemonset-6jr69" Jul 28 02:10:54.972437 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:54.972396 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e1ddcbd2-35c4-4177-88bd-4086374dc0e3-podres\") pod \"perf-node-gather-daemonset-6jr69\" (UID: \"e1ddcbd2-35c4-4177-88bd-4086374dc0e3\") " pod="openshift-must-gather-wvjdf/perf-node-gather-daemonset-6jr69" Jul 28 02:10:54.974807 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:54.974786 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wvjdf\"/\"kube-root-ca.crt\"" Jul 28 02:10:54.975748 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:54.975728 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wvjdf\"/\"default-dockercfg-njkfq\"" Jul 28 02:10:54.975852 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:54.975733 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wvjdf\"/\"openshift-service-ca.crt\"" Jul 28 02:10:54.982374 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:54.982349 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wvjdf/perf-node-gather-daemonset-6jr69"] Jul 28 02:10:55.073270 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:55.073231 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e1ddcbd2-35c4-4177-88bd-4086374dc0e3-proc\") pod \"perf-node-gather-daemonset-6jr69\" (UID: \"e1ddcbd2-35c4-4177-88bd-4086374dc0e3\") " pod="openshift-must-gather-wvjdf/perf-node-gather-daemonset-6jr69" Jul 28 02:10:55.073467 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:55.073279 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1ddcbd2-35c4-4177-88bd-4086374dc0e3-sys\") pod \"perf-node-gather-daemonset-6jr69\" (UID: \"e1ddcbd2-35c4-4177-88bd-4086374dc0e3\") " pod="openshift-must-gather-wvjdf/perf-node-gather-daemonset-6jr69" Jul 28 02:10:55.073467 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:55.073302 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sf27\" (UniqueName: \"kubernetes.io/projected/e1ddcbd2-35c4-4177-88bd-4086374dc0e3-kube-api-access-7sf27\") pod \"perf-node-gather-daemonset-6jr69\" (UID: \"e1ddcbd2-35c4-4177-88bd-4086374dc0e3\") " pod="openshift-must-gather-wvjdf/perf-node-gather-daemonset-6jr69" Jul 28 02:10:55.073467 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:55.073341 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e1ddcbd2-35c4-4177-88bd-4086374dc0e3-podres\") pod \"perf-node-gather-daemonset-6jr69\" (UID: \"e1ddcbd2-35c4-4177-88bd-4086374dc0e3\") " pod="openshift-must-gather-wvjdf/perf-node-gather-daemonset-6jr69" Jul 28 02:10:55.073467 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:55.073345 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e1ddcbd2-35c4-4177-88bd-4086374dc0e3-proc\") pod \"perf-node-gather-daemonset-6jr69\" (UID: \"e1ddcbd2-35c4-4177-88bd-4086374dc0e3\") " pod="openshift-must-gather-wvjdf/perf-node-gather-daemonset-6jr69" Jul 28 02:10:55.073467 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:55.073397 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e1ddcbd2-35c4-4177-88bd-4086374dc0e3-lib-modules\") pod \"perf-node-gather-daemonset-6jr69\" (UID: \"e1ddcbd2-35c4-4177-88bd-4086374dc0e3\") " pod="openshift-must-gather-wvjdf/perf-node-gather-daemonset-6jr69" Jul 28 02:10:55.073467 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:55.073399 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1ddcbd2-35c4-4177-88bd-4086374dc0e3-sys\") pod \"perf-node-gather-daemonset-6jr69\" (UID: \"e1ddcbd2-35c4-4177-88bd-4086374dc0e3\") " pod="openshift-must-gather-wvjdf/perf-node-gather-daemonset-6jr69" Jul 28 02:10:55.073467 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:55.073455 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e1ddcbd2-35c4-4177-88bd-4086374dc0e3-podres\") pod \"perf-node-gather-daemonset-6jr69\" (UID: \"e1ddcbd2-35c4-4177-88bd-4086374dc0e3\") " pod="openshift-must-gather-wvjdf/perf-node-gather-daemonset-6jr69" Jul 28 02:10:55.073751 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:55.073475 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e1ddcbd2-35c4-4177-88bd-4086374dc0e3-lib-modules\") pod \"perf-node-gather-daemonset-6jr69\" (UID: \"e1ddcbd2-35c4-4177-88bd-4086374dc0e3\") " pod="openshift-must-gather-wvjdf/perf-node-gather-daemonset-6jr69" Jul 28 02:10:55.082797 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:55.082770 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sf27\" (UniqueName: \"kubernetes.io/projected/e1ddcbd2-35c4-4177-88bd-4086374dc0e3-kube-api-access-7sf27\") pod \"perf-node-gather-daemonset-6jr69\" (UID: \"e1ddcbd2-35c4-4177-88bd-4086374dc0e3\") " pod="openshift-must-gather-wvjdf/perf-node-gather-daemonset-6jr69" Jul 28 02:10:55.218175 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:55.218086 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-m8jjv_fa68b140-0e24-4f1c-a9c3-7125ce3046da/console-operator/0.log" Jul 28 02:10:55.281880 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:55.281851 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wvjdf/perf-node-gather-daemonset-6jr69" Jul 28 02:10:55.414429 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:55.414401 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wvjdf/perf-node-gather-daemonset-6jr69"] Jul 28 02:10:55.416206 ip-10-0-131-23 kubenswrapper[2565]: W0728 02:10:55.416178 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode1ddcbd2_35c4_4177_88bd_4086374dc0e3.slice/crio-4ad551b48beb94f83eac9052a7f93a00527df88c2e9497ebb06621f0ac127f79 WatchSource:0}: Error finding container 4ad551b48beb94f83eac9052a7f93a00527df88c2e9497ebb06621f0ac127f79: Status 404 returned error can't find the container with id 4ad551b48beb94f83eac9052a7f93a00527df88c2e9497ebb06621f0ac127f79 Jul 28 02:10:55.440463 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:55.440437 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wvjdf/perf-node-gather-daemonset-6jr69" event={"ID":"e1ddcbd2-35c4-4177-88bd-4086374dc0e3","Type":"ContainerStarted","Data":"4ad551b48beb94f83eac9052a7f93a00527df88c2e9497ebb06621f0ac127f79"} Jul 28 02:10:56.447645 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:56.447606 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wvjdf/perf-node-gather-daemonset-6jr69" event={"ID":"e1ddcbd2-35c4-4177-88bd-4086374dc0e3","Type":"ContainerStarted","Data":"82037c5977c75baf0cd4ae1cebb04edb051ff1aa8b8428ab5bee77f53bd31557"} Jul 28 02:10:56.448131 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:56.447692 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wvjdf/perf-node-gather-daemonset-6jr69" Jul 28 02:10:56.464618 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:56.464566 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wvjdf/perf-node-gather-daemonset-6jr69" podStartSLOduration=2.46455087 podStartE2EDuration="2.46455087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 02:10:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:10:56.462360847 +0000 UTC m=+3313.896759166" watchObservedRunningTime="2026-07-28 02:10:56.46455087 +0000 UTC m=+3313.898949185" Jul 28 02:10:57.573917 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:57.573886 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ftbqh_dadc10df-28cc-4323-90c4-759cc38b55a7/dns/0.log" Jul 28 02:10:57.593661 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:57.593633 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ftbqh_dadc10df-28cc-4323-90c4-759cc38b55a7/kube-rbac-proxy/0.log" Jul 28 02:10:57.690713 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:57.690684 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-snwqc_a97a9823-8b73-453c-9d16-8907cc6b5116/dns-node-resolver/0.log" Jul 28 02:10:58.241925 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:58.241884 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6c8c86447b-8tdk7_af734a7d-7584-4a47-b67a-f0b0ecfd551b/registry/0.log" Jul 28 02:10:58.327025 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:58.326981 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zlwvl_f32e8846-0caa-4b52-a5fd-ae65659fdea1/node-ca/0.log" Jul 28 02:10:59.292520 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:59.292492 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj_d56a28fd-a428-4306-95a9-065e8e92e8bc/istio-proxy/0.log" Jul 28 02:10:59.563541 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:10:59.563460 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-85hcz_15fcf533-e2cd-4e92-8891-1d27b46f6ea9/discovery/0.log" Jul 28 02:11:00.649046 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:11:00.649004 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fdcrp_88f7e7b8-110e-472b-ba6c-d2a612daaf07/serve-healthcheck-canary/0.log" Jul 28 02:11:01.313324 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:11:01.313292 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w6xtp_72e36b06-6877-4705-9477-4682cd601a21/kube-rbac-proxy/0.log" Jul 28 02:11:01.332562 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:11:01.332538 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w6xtp_72e36b06-6877-4705-9477-4682cd601a21/exporter/0.log" Jul 28 02:11:01.352959 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:11:01.352920 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w6xtp_72e36b06-6877-4705-9477-4682cd601a21/extractor/0.log" Jul 28 02:11:02.462533 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:11:02.462506 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wvjdf/perf-node-gather-daemonset-6jr69" Jul 28 02:11:04.356414 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:11:04.356370 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8699bb9d7c-6p952_867b0a1e-9249-4fbc-be63-e0495f3aff8b/manager/0.log" Jul 28 02:11:04.377330 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:11:04.377306 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8699bb9d7c-6p952_867b0a1e-9249-4fbc-be63-e0495f3aff8b/copy-manifests/0.log" Jul 28 02:11:05.334872 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:11:05.334835 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj_d56a28fd-a428-4306-95a9-065e8e92e8bc/istio-proxy/0.log" Jul 28 02:11:05.597266 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:11:05.597233 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-85hcz_15fcf533-e2cd-4e92-8891-1d27b46f6ea9/discovery/0.log" Jul 28 02:11:05.765270 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:11:05.765240 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7d7cccf554-p9gjs_9f3cab44-ce9b-4d19-82ec-d47a0876b021/manager/0.log" Jul 28 02:11:07.151050 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:11:07.150943 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cd9qdj_d56a28fd-a428-4306-95a9-065e8e92e8bc/istio-proxy/0.log" Jul 28 02:11:07.415573 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:11:07.415499 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-85hcz_15fcf533-e2cd-4e92-8891-1d27b46f6ea9/discovery/0.log" Jul 28 02:11:12.218419 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:11:12.218390 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2trzf_a29b525f-58ac-47f8-b759-d81742c281f4/kube-multus-additional-cni-plugins/0.log" Jul 28 02:11:12.240447 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:11:12.240418 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2trzf_a29b525f-58ac-47f8-b759-d81742c281f4/egress-router-binary-copy/0.log" Jul 28 02:11:12.262578 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:11:12.262546 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2trzf_a29b525f-58ac-47f8-b759-d81742c281f4/cni-plugins/0.log" Jul 28 02:11:12.282679 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:11:12.282654 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2trzf_a29b525f-58ac-47f8-b759-d81742c281f4/bond-cni-plugin/0.log" Jul 28 02:11:12.300704 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:11:12.300675 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2trzf_a29b525f-58ac-47f8-b759-d81742c281f4/routeoverride-cni/0.log" Jul 28 02:11:12.319987 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:11:12.319946 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2trzf_a29b525f-58ac-47f8-b759-d81742c281f4/whereabouts-cni-bincopy/0.log" Jul 28 02:11:12.341689 ip-10-0-131-23 kubenswrapper[2565]: I0728 02:11:12.341661 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2trzf_a29b525f-58ac-47f8-b759-d81742c281f4/whereabouts-cni/0.log"