Aug 04 15:54:37.386670 ip-10-0-141-210 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Aug 04 15:54:37.386682 ip-10-0-141-210 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Aug 04 15:54:37.386689 ip-10-0-141-210 systemd[1]: kubelet.service: Failed with result 'resources'. Aug 04 15:54:37.386940 ip-10-0-141-210 systemd[1]: Failed to start Kubernetes Kubelet. Aug 04 15:54:47.770872 ip-10-0-141-210 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Aug 04 15:54:47.770885 ip-10-0-141-210 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 535aff6149b14c7a965f89d13589ee87 -- Aug 04 15:56:40.470850 ip-10-0-141-210 systemd[1]: Starting Kubernetes Kubelet... Aug 04 15:56:40.853287 ip-10-0-141-210 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 15:56:40.853287 ip-10-0-141-210 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Aug 04 15:56:40.853287 ip-10-0-141-210 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 15:56:40.853287 ip-10-0-141-210 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Aug 04 15:56:40.853287 ip-10-0-141-210 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 15:56:40.854027 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.853343 2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Aug 04 15:56:40.856829 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856813 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 15:56:40.856829 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856828 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 15:56:40.856829 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856832 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 15:56:40.856922 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856836 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 15:56:40.856922 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856839 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 15:56:40.856922 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856842 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 15:56:40.856922 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856844 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 15:56:40.856922 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856847 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 15:56:40.856922 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856850 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 15:56:40.856922 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856853 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 15:56:40.856922 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856856 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 15:56:40.856922 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856858 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 15:56:40.856922 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856861 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 15:56:40.856922 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856863 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 15:56:40.856922 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856866 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 15:56:40.856922 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856868 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 15:56:40.856922 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856871 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 15:56:40.856922 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856873 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 15:56:40.856922 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856876 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 15:56:40.856922 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856878 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 15:56:40.856922 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856881 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 15:56:40.856922 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856888 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 15:56:40.856922 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856891 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 15:56:40.857360 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856893 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 15:56:40.857360 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856895 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 15:56:40.857360 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856898 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 15:56:40.857360 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856900 2573 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 15:56:40.857360 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856904 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 15:56:40.857360 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856907 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 15:56:40.857360 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856910 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 15:56:40.857360 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856912 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 15:56:40.857360 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856915 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 15:56:40.857360 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856919 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 15:56:40.857360 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856922 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 15:56:40.857360 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856925 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 15:56:40.857360 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856928 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 15:56:40.857360 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856930 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 15:56:40.857360 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856933 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 15:56:40.857360 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856935 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 15:56:40.857360 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856938 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 15:56:40.857360 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856940 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 15:56:40.857360 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856943 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 15:56:40.857360 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856945 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 15:56:40.857868 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856947 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 15:56:40.857868 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856950 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 15:56:40.857868 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856952 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 15:56:40.857868 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856954 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 15:56:40.857868 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856957 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 15:56:40.857868 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856959 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 15:56:40.857868 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856962 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 15:56:40.857868 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856964 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 15:56:40.857868 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856966 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 15:56:40.857868 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856969 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 15:56:40.857868 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856971 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 15:56:40.857868 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856973 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 15:56:40.857868 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856976 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 15:56:40.857868 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856979 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 15:56:40.857868 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856981 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 15:56:40.857868 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856984 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 15:56:40.857868 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856986 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 15:56:40.857868 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856989 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 15:56:40.857868 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856991 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 15:56:40.857868 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856993 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 15:56:40.858315 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856996 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 15:56:40.858315 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.856998 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 15:56:40.858315 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857000 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 15:56:40.858315 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857003 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 15:56:40.858315 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857005 2573 feature_gate.go:328] unrecognized feature gate: Example Aug 04 15:56:40.858315 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857007 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 15:56:40.858315 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857010 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 15:56:40.858315 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857012 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 15:56:40.858315 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857014 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 15:56:40.858315 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857016 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 15:56:40.858315 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857019 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 15:56:40.858315 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857021 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 15:56:40.858315 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857026 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 15:56:40.858315 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857029 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 15:56:40.858315 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857032 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 15:56:40.858315 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857035 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 15:56:40.858315 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857037 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 15:56:40.858315 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857040 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 15:56:40.858315 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857043 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 15:56:40.858766 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857045 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 15:56:40.858766 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857048 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 15:56:40.858766 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857050 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 15:56:40.858766 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857052 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 15:56:40.858766 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857055 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 15:56:40.858766 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857058 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 15:56:40.858766 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857062 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 15:56:40.858766 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857064 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 15:56:40.858766 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857067 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 15:56:40.858766 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857069 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 15:56:40.858766 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857071 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 15:56:40.858766 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857074 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 15:56:40.858766 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857076 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 15:56:40.858766 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857078 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 15:56:40.858766 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857081 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 15:56:40.858766 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857083 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 15:56:40.858766 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857085 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 15:56:40.858766 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857743 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 15:56:40.858766 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857749 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 15:56:40.859182 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857751 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 15:56:40.859182 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857754 2573 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 15:56:40.859182 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857757 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 15:56:40.859182 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857759 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 15:56:40.859182 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857767 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 15:56:40.859182 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857770 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 15:56:40.859182 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857773 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 15:56:40.859182 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857775 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 15:56:40.859182 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857778 2573 feature_gate.go:328] unrecognized feature gate: Example Aug 04 15:56:40.859182 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857780 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 15:56:40.859182 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857783 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 15:56:40.859182 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857786 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 15:56:40.859182 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857788 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 15:56:40.859182 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857790 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 15:56:40.859182 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857793 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 15:56:40.859182 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857795 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 15:56:40.859182 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857797 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 15:56:40.859182 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857800 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 15:56:40.859182 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857802 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 15:56:40.859182 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857806 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 15:56:40.859824 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857809 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 15:56:40.859824 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857812 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 15:56:40.859824 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857814 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 15:56:40.859824 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857816 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 15:56:40.859824 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857819 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 15:56:40.859824 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857821 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 15:56:40.859824 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857823 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 15:56:40.859824 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857826 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 15:56:40.859824 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857828 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 15:56:40.859824 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857830 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 15:56:40.859824 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857833 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 15:56:40.859824 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857835 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 15:56:40.859824 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857837 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 15:56:40.859824 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857841 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 15:56:40.859824 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857844 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 15:56:40.859824 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857847 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 15:56:40.859824 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857849 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 15:56:40.859824 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857852 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 15:56:40.859824 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857854 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 15:56:40.859824 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857856 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 15:56:40.860570 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857859 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 15:56:40.860570 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857862 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 15:56:40.860570 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857864 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 15:56:40.860570 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857868 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 15:56:40.860570 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857871 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 15:56:40.860570 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857874 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 15:56:40.860570 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857877 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 15:56:40.860570 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857880 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 15:56:40.860570 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857882 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 15:56:40.860570 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857885 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 15:56:40.860570 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857887 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 15:56:40.860570 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857891 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 15:56:40.860570 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857894 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 15:56:40.860570 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857896 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 15:56:40.860570 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857898 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 15:56:40.860570 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857901 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 15:56:40.860570 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857904 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 15:56:40.860570 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857907 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 15:56:40.860570 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857910 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 15:56:40.861056 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857913 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 15:56:40.861056 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857915 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 15:56:40.861056 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857917 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 15:56:40.861056 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857920 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 15:56:40.861056 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857922 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 15:56:40.861056 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857924 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 15:56:40.861056 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857927 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 15:56:40.861056 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857929 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 15:56:40.861056 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857931 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 15:56:40.861056 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857933 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 15:56:40.861056 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857936 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 15:56:40.861056 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857938 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 15:56:40.861056 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857940 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 15:56:40.861056 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857942 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 15:56:40.861056 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857945 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 15:56:40.861056 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857947 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 15:56:40.861056 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857951 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 15:56:40.861056 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857953 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 15:56:40.861056 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857956 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 15:56:40.861488 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857958 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 15:56:40.861488 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857960 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 15:56:40.861488 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857963 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 15:56:40.861488 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857965 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 15:56:40.861488 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857967 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 15:56:40.861488 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857970 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 15:56:40.861488 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857973 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 15:56:40.861488 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857975 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 15:56:40.861488 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857978 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 15:56:40.861488 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857980 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 15:56:40.861488 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857982 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 15:56:40.861488 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857984 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 15:56:40.861488 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857987 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 15:56:40.861488 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857989 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 15:56:40.861488 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857991 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 15:56:40.861488 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857993 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 15:56:40.861488 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857996 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 15:56:40.861488 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.857998 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 15:56:40.861488 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858000 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 15:56:40.861488 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858339 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 15:56:40.861980 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858343 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 15:56:40.861980 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858345 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 15:56:40.861980 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858348 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 15:56:40.861980 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858350 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 15:56:40.861980 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858352 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 15:56:40.861980 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858355 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 15:56:40.861980 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858358 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 15:56:40.861980 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858361 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 15:56:40.861980 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858366 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 15:56:40.861980 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858368 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 15:56:40.861980 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858370 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 15:56:40.861980 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858373 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 15:56:40.861980 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858375 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 15:56:40.861980 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858378 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 15:56:40.861980 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858381 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 15:56:40.861980 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858383 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 15:56:40.861980 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858386 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 15:56:40.861980 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858388 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 15:56:40.861980 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858391 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 15:56:40.861980 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858393 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 15:56:40.862442 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858396 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 15:56:40.862442 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858398 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 15:56:40.862442 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858401 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 15:56:40.862442 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858405 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 15:56:40.862442 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858408 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 15:56:40.862442 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858411 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 15:56:40.862442 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858414 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 15:56:40.862442 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858416 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 15:56:40.862442 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858419 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 15:56:40.862442 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858421 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 15:56:40.862442 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858424 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 15:56:40.862442 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858427 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 15:56:40.862442 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858431 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 15:56:40.862442 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858433 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 15:56:40.862442 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858436 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 15:56:40.862442 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858439 2573 feature_gate.go:328] unrecognized feature gate: Example Aug 04 15:56:40.862442 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858441 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 15:56:40.862442 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858444 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 15:56:40.862442 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858447 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 15:56:40.862442 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858449 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 15:56:40.862920 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858453 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 15:56:40.862920 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858455 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 15:56:40.862920 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858458 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 15:56:40.862920 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858460 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 15:56:40.862920 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858472 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 15:56:40.862920 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858474 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 15:56:40.862920 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858477 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 15:56:40.862920 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858479 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 15:56:40.862920 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858482 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 15:56:40.862920 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858484 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 15:56:40.862920 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858487 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 15:56:40.862920 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858489 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 15:56:40.862920 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858492 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 15:56:40.862920 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858494 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 15:56:40.862920 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858497 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 15:56:40.862920 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858499 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 15:56:40.862920 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858502 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 15:56:40.862920 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858504 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 15:56:40.862920 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858506 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 15:56:40.863353 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858509 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 15:56:40.863353 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858511 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 15:56:40.863353 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858514 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 15:56:40.863353 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858516 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 15:56:40.863353 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858520 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 15:56:40.863353 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858523 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 15:56:40.863353 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858526 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 15:56:40.863353 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858528 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 15:56:40.863353 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858531 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 15:56:40.863353 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858533 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 15:56:40.863353 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858536 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 15:56:40.863353 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858538 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 15:56:40.863353 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858540 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 15:56:40.863353 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858558 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 15:56:40.863353 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858561 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 15:56:40.863353 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858564 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 15:56:40.863353 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858566 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 15:56:40.863353 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858570 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 15:56:40.863353 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858573 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 15:56:40.863798 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858575 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 15:56:40.863798 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858578 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 15:56:40.863798 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858580 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 15:56:40.863798 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858583 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 15:56:40.863798 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858586 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 15:56:40.863798 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858589 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 15:56:40.863798 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858592 2573 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 15:56:40.863798 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858595 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 15:56:40.863798 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858598 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 15:56:40.863798 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858601 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 15:56:40.863798 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858603 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 15:56:40.863798 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858606 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 15:56:40.863798 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858608 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 15:56:40.863798 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858610 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 15:56:40.863798 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858613 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 15:56:40.863798 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858615 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 15:56:40.863798 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858617 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 15:56:40.863798 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858620 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 15:56:40.863798 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858622 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 15:56:40.863798 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.858624 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 15:56:40.864246 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858698 2573 flags.go:64] FLAG: --address="0.0.0.0" Aug 04 15:56:40.864246 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858715 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Aug 04 15:56:40.864246 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858727 2573 flags.go:64] FLAG: --anonymous-auth="true" Aug 04 15:56:40.864246 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858733 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Aug 04 15:56:40.864246 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858738 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Aug 04 15:56:40.864246 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858741 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Aug 04 15:56:40.864246 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858746 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Aug 04 15:56:40.864246 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858751 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Aug 04 15:56:40.864246 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858754 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Aug 04 15:56:40.864246 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858756 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Aug 04 15:56:40.864246 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858761 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Aug 04 15:56:40.864246 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858764 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Aug 04 15:56:40.864246 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858767 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Aug 04 15:56:40.864246 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858770 2573 flags.go:64] FLAG: --cgroup-root="" Aug 04 15:56:40.864246 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858772 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Aug 04 15:56:40.864246 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858775 2573 flags.go:64] FLAG: --client-ca-file="" Aug 04 15:56:40.864246 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858778 2573 flags.go:64] FLAG: --cloud-provider="external" Aug 04 15:56:40.864246 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858781 2573 flags.go:64] FLAG: --cluster-dns="[]" Aug 04 15:56:40.864246 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858785 2573 flags.go:64] FLAG: --cluster-domain="" Aug 04 15:56:40.864246 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858788 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Aug 04 15:56:40.864246 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858792 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Aug 04 15:56:40.864246 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858795 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Aug 04 15:56:40.864246 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858798 2573 flags.go:64] FLAG: --container-log-max-files="5" Aug 04 15:56:40.864246 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858801 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Aug 04 15:56:40.864788 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858804 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Aug 04 15:56:40.864788 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858807 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Aug 04 15:56:40.864788 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858810 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Aug 04 15:56:40.864788 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858813 2573 flags.go:64] FLAG: --contention-profiling="false" Aug 04 15:56:40.864788 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858816 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Aug 04 15:56:40.864788 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858818 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Aug 04 15:56:40.864788 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858821 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Aug 04 15:56:40.864788 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858824 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Aug 04 15:56:40.864788 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858827 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Aug 04 15:56:40.864788 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858830 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Aug 04 15:56:40.864788 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858832 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Aug 04 15:56:40.864788 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858835 2573 flags.go:64] FLAG: --enable-load-reader="false" Aug 04 15:56:40.864788 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858838 2573 flags.go:64] FLAG: --enable-server="true" Aug 04 15:56:40.864788 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858840 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Aug 04 15:56:40.864788 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858846 2573 flags.go:64] FLAG: --event-burst="100" Aug 04 15:56:40.864788 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858849 2573 flags.go:64] FLAG: --event-qps="50" Aug 04 15:56:40.864788 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858851 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Aug 04 15:56:40.864788 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858854 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Aug 04 15:56:40.864788 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858858 2573 flags.go:64] FLAG: --eviction-hard="" Aug 04 15:56:40.864788 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858861 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Aug 04 15:56:40.864788 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858864 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Aug 04 15:56:40.864788 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858867 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Aug 04 15:56:40.864788 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858870 2573 flags.go:64] FLAG: --eviction-soft="" Aug 04 15:56:40.864788 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858872 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Aug 04 15:56:40.864788 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858875 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Aug 04 15:56:40.865395 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858878 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Aug 04 15:56:40.865395 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858881 2573 flags.go:64] FLAG: --experimental-mounter-path="" Aug 04 15:56:40.865395 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858883 2573 flags.go:64] FLAG: --fail-cgroupv1="false" Aug 04 15:56:40.865395 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858886 2573 flags.go:64] FLAG: --fail-swap-on="true" Aug 04 15:56:40.865395 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858889 2573 flags.go:64] FLAG: --feature-gates="" Aug 04 15:56:40.865395 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858893 2573 flags.go:64] FLAG: --file-check-frequency="20s" Aug 04 15:56:40.865395 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858895 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Aug 04 15:56:40.865395 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858898 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Aug 04 15:56:40.865395 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858901 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Aug 04 15:56:40.865395 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858904 2573 flags.go:64] FLAG: --healthz-port="10248" Aug 04 15:56:40.865395 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858907 2573 flags.go:64] FLAG: --help="false" Aug 04 15:56:40.865395 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858910 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-141-210.ec2.internal" Aug 04 15:56:40.865395 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858913 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Aug 04 15:56:40.865395 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858915 2573 flags.go:64] FLAG: --http-check-frequency="20s" Aug 04 15:56:40.865395 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858918 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Aug 04 15:56:40.865395 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858921 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Aug 04 15:56:40.865395 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858924 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Aug 04 15:56:40.865395 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858927 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Aug 04 15:56:40.865395 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858929 2573 flags.go:64] FLAG: --image-service-endpoint="" Aug 04 15:56:40.865395 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858932 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Aug 04 15:56:40.865395 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858935 2573 flags.go:64] FLAG: --kube-api-burst="100" Aug 04 15:56:40.865395 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858938 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Aug 04 15:56:40.865395 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858941 2573 flags.go:64] FLAG: --kube-api-qps="50" Aug 04 15:56:40.865395 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858944 2573 flags.go:64] FLAG: --kube-reserved="" Aug 04 15:56:40.865964 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858947 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Aug 04 15:56:40.865964 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858950 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Aug 04 15:56:40.865964 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858953 2573 flags.go:64] FLAG: --kubelet-cgroups="" Aug 04 15:56:40.865964 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858956 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Aug 04 15:56:40.865964 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858959 2573 flags.go:64] FLAG: --lock-file="" Aug 04 15:56:40.865964 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858961 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Aug 04 15:56:40.865964 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858964 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Aug 04 15:56:40.865964 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858966 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Aug 04 15:56:40.865964 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858971 2573 flags.go:64] FLAG: --log-json-split-stream="false" Aug 04 15:56:40.865964 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858974 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Aug 04 15:56:40.865964 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858977 2573 flags.go:64] FLAG: --log-text-split-stream="false" Aug 04 15:56:40.865964 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858979 2573 flags.go:64] FLAG: --logging-format="text" Aug 04 15:56:40.865964 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858982 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Aug 04 15:56:40.865964 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858986 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Aug 04 15:56:40.865964 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858988 2573 flags.go:64] FLAG: --manifest-url="" Aug 04 15:56:40.865964 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858991 2573 flags.go:64] FLAG: --manifest-url-header="" Aug 04 15:56:40.865964 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858995 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Aug 04 15:56:40.865964 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.858998 2573 flags.go:64] FLAG: --max-open-files="1000000" Aug 04 15:56:40.865964 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859002 2573 flags.go:64] FLAG: --max-pods="110" Aug 04 15:56:40.865964 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859004 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Aug 04 15:56:40.865964 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859007 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Aug 04 15:56:40.865964 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859010 2573 flags.go:64] FLAG: --memory-manager-policy="None" Aug 04 15:56:40.865964 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859013 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Aug 04 15:56:40.865964 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859015 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Aug 04 15:56:40.865964 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859018 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Aug 04 15:56:40.866532 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859021 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Aug 04 15:56:40.866532 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859028 2573 flags.go:64] FLAG: --node-status-max-images="50" Aug 04 15:56:40.866532 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859031 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Aug 04 15:56:40.866532 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859034 2573 flags.go:64] FLAG: --oom-score-adj="-999" Aug 04 15:56:40.866532 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859037 2573 flags.go:64] FLAG: --pod-cidr="" Aug 04 15:56:40.866532 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859040 2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Aug 04 15:56:40.866532 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859045 2573 flags.go:64] FLAG: --pod-manifest-path="" Aug 04 15:56:40.866532 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859048 2573 flags.go:64] FLAG: --pod-max-pids="-1" Aug 04 15:56:40.866532 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859051 2573 flags.go:64] FLAG: --pods-per-core="0" Aug 04 15:56:40.866532 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859055 2573 flags.go:64] FLAG: --port="10250" Aug 04 15:56:40.866532 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859058 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Aug 04 15:56:40.866532 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859061 2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08d8a3c54e7a42f7e" Aug 04 15:56:40.866532 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859064 2573 flags.go:64] FLAG: --qos-reserved="" Aug 04 15:56:40.866532 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859067 2573 flags.go:64] FLAG: --read-only-port="10255" Aug 04 15:56:40.866532 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859070 2573 flags.go:64] FLAG: --register-node="true" Aug 04 15:56:40.866532 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859072 2573 flags.go:64] FLAG: --register-with-taints="" Aug 04 15:56:40.866532 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859076 2573 flags.go:64] FLAG: --registry-burst="10" Aug 04 15:56:40.866532 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859078 2573 flags.go:64] FLAG: --registry-qps="5" Aug 04 15:56:40.866532 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859081 2573 flags.go:64] FLAG: --reserved-cpus="" Aug 04 15:56:40.866532 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859084 2573 flags.go:64] FLAG: --reserved-memory="" Aug 04 15:56:40.866532 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859087 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Aug 04 15:56:40.866532 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859090 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Aug 04 15:56:40.866532 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859093 2573 flags.go:64] FLAG: --rotate-certificates="false" Aug 04 15:56:40.866532 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859096 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Aug 04 15:56:40.866532 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859098 2573 flags.go:64] FLAG: --runonce="false" Aug 04 15:56:40.867124 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859101 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Aug 04 15:56:40.867124 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859104 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Aug 04 15:56:40.867124 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859107 2573 flags.go:64] FLAG: --seccomp-default="false" Aug 04 15:56:40.867124 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859110 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Aug 04 15:56:40.867124 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859112 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Aug 04 15:56:40.867124 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859115 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Aug 04 15:56:40.867124 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859118 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Aug 04 15:56:40.867124 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859121 2573 flags.go:64] FLAG: --storage-driver-password="root" Aug 04 15:56:40.867124 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859123 2573 flags.go:64] FLAG: --storage-driver-secure="false" Aug 04 15:56:40.867124 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859126 2573 flags.go:64] FLAG: --storage-driver-table="stats" Aug 04 15:56:40.867124 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859129 2573 flags.go:64] FLAG: --storage-driver-user="root" Aug 04 15:56:40.867124 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859131 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Aug 04 15:56:40.867124 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859134 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Aug 04 15:56:40.867124 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859136 2573 flags.go:64] FLAG: --system-cgroups="" Aug 04 15:56:40.867124 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859139 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Aug 04 15:56:40.867124 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859144 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Aug 04 15:56:40.867124 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859148 2573 flags.go:64] FLAG: --tls-cert-file="" Aug 04 15:56:40.867124 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859150 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Aug 04 15:56:40.867124 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859154 2573 flags.go:64] FLAG: --tls-min-version="" Aug 04 15:56:40.867124 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859157 2573 flags.go:64] FLAG: --tls-private-key-file="" Aug 04 15:56:40.867124 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859159 2573 flags.go:64] FLAG: --topology-manager-policy="none" Aug 04 15:56:40.867124 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859162 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Aug 04 15:56:40.867124 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859164 2573 flags.go:64] FLAG: --topology-manager-scope="container" Aug 04 15:56:40.867124 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859167 2573 flags.go:64] FLAG: --v="2" Aug 04 15:56:40.867124 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859171 2573 flags.go:64] FLAG: --version="false" Aug 04 15:56:40.867702 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859175 2573 flags.go:64] FLAG: --vmodule="" Aug 04 15:56:40.867702 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859178 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Aug 04 15:56:40.867702 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.859182 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Aug 04 15:56:40.867702 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.859290 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 15:56:40.867702 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.859338 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 15:56:40.867702 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860355 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 15:56:40.867702 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860365 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 15:56:40.867702 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860368 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 15:56:40.867702 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860373 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 15:56:40.867702 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860377 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 15:56:40.867702 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860384 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 15:56:40.867702 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860390 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 15:56:40.867702 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860398 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 15:56:40.867702 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860401 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 15:56:40.867702 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860403 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 15:56:40.867702 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860406 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 15:56:40.867702 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860409 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 15:56:40.867702 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860412 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 15:56:40.867702 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860414 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 15:56:40.867702 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860419 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 15:56:40.868162 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860423 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 15:56:40.868162 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860429 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 15:56:40.868162 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860447 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 15:56:40.868162 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860452 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 15:56:40.868162 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860463 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 15:56:40.868162 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860475 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 15:56:40.868162 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860480 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 15:56:40.868162 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860484 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 15:56:40.868162 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860488 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 15:56:40.868162 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860492 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 15:56:40.868162 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860496 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 15:56:40.868162 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860500 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 15:56:40.868162 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860504 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 15:56:40.868162 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860507 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 15:56:40.868162 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860514 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 15:56:40.868162 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860518 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 15:56:40.868162 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860526 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 15:56:40.868162 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860532 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 15:56:40.868162 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860536 2573 feature_gate.go:328] unrecognized feature gate: Example Aug 04 15:56:40.868600 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860540 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 15:56:40.868600 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860559 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 15:56:40.868600 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860563 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 15:56:40.868600 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860567 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 15:56:40.868600 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860570 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 15:56:40.868600 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860574 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 15:56:40.868600 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860578 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 15:56:40.868600 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860582 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 15:56:40.868600 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860586 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 15:56:40.868600 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860590 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 15:56:40.868600 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860598 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 15:56:40.868600 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860602 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 15:56:40.868600 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860606 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 15:56:40.868600 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860610 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 15:56:40.868600 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860614 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 15:56:40.868600 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860618 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 15:56:40.868600 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860622 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 15:56:40.868600 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860626 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 15:56:40.868600 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860630 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 15:56:40.868600 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860634 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 15:56:40.869087 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860637 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 15:56:40.869087 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860641 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 15:56:40.869087 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860645 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 15:56:40.869087 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860653 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 15:56:40.869087 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860657 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 15:56:40.869087 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860662 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 15:56:40.869087 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860668 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 15:56:40.869087 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860680 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 15:56:40.869087 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860693 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 15:56:40.869087 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860698 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 15:56:40.869087 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860702 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 15:56:40.869087 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860845 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 15:56:40.869087 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860850 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 15:56:40.869087 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860854 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 15:56:40.869087 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860857 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 15:56:40.869087 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860860 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 15:56:40.869087 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860863 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 15:56:40.869087 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860866 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 15:56:40.869087 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860868 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 15:56:40.869087 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860870 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 15:56:40.869563 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860873 2573 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 15:56:40.869563 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860876 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 15:56:40.869563 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860878 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 15:56:40.869563 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860881 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 15:56:40.869563 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860883 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 15:56:40.869563 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860886 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 15:56:40.869563 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860889 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 15:56:40.869563 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860891 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 15:56:40.869563 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860894 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 15:56:40.869563 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860896 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 15:56:40.869563 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860899 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 15:56:40.869563 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860901 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 15:56:40.869563 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860905 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 15:56:40.869563 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860909 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 15:56:40.869563 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860911 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 15:56:40.869563 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860914 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 15:56:40.869563 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860917 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 15:56:40.869563 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860919 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 15:56:40.869563 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860922 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 15:56:40.869563 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860924 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 15:56:40.870032 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860927 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 15:56:40.870032 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860929 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 15:56:40.870032 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.860932 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 15:56:40.870032 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.861544 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 15:56:40.870032 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.868247 2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Aug 04 15:56:40.870032 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.868263 2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Aug 04 15:56:40.870032 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.868994 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Aug 04 15:56:40.870032 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.869002 2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Aug 04 15:56:40.870032 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869054 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 15:56:40.870032 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869058 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 15:56:40.870032 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869061 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 15:56:40.870032 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869064 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 15:56:40.870032 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869068 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 15:56:40.870032 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869071 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 15:56:40.870032 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869074 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 15:56:40.870379 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869077 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 15:56:40.870379 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869079 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 15:56:40.870379 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869082 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 15:56:40.870379 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869084 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 15:56:40.870379 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869087 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 15:56:40.870379 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869090 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 15:56:40.870379 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869093 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 15:56:40.870379 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869095 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 15:56:40.870379 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869099 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 15:56:40.870379 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869103 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 15:56:40.870379 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869106 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 15:56:40.870379 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869109 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 15:56:40.870379 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869111 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 15:56:40.870379 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869114 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 15:56:40.870379 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869117 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 15:56:40.870379 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869119 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 15:56:40.870379 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869122 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 15:56:40.870379 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869124 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 15:56:40.870379 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869127 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 15:56:40.870826 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869129 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 15:56:40.870826 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869132 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 15:56:40.870826 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869135 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 15:56:40.870826 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869141 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 15:56:40.870826 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869144 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 15:56:40.870826 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869147 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 15:56:40.870826 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869150 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 15:56:40.870826 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869153 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 15:56:40.870826 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869158 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 15:56:40.870826 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869160 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 15:56:40.870826 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869163 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 15:56:40.870826 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869165 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 15:56:40.870826 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869168 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 15:56:40.870826 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869170 2573 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 15:56:40.870826 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869173 2573 feature_gate.go:328] unrecognized feature gate: Example Aug 04 15:56:40.870826 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869175 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 15:56:40.870826 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869177 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 15:56:40.870826 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869180 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 15:56:40.870826 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869182 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 15:56:40.870826 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869184 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 15:56:40.871274 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869187 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 15:56:40.871274 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869189 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 15:56:40.871274 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869192 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 15:56:40.871274 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869194 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 15:56:40.871274 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869197 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 15:56:40.871274 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869199 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 15:56:40.871274 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869201 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 15:56:40.871274 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869204 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 15:56:40.871274 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869206 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 15:56:40.871274 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869209 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 15:56:40.871274 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869211 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 15:56:40.871274 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869214 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 15:56:40.871274 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869216 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 15:56:40.871274 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869218 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 15:56:40.871274 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869221 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 15:56:40.871274 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869223 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 15:56:40.871274 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869226 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 15:56:40.871274 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869229 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 15:56:40.871274 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869231 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 15:56:40.871274 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869234 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 15:56:40.871745 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869236 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 15:56:40.871745 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869238 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 15:56:40.871745 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869241 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 15:56:40.871745 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869243 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 15:56:40.871745 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869245 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 15:56:40.871745 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869248 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 15:56:40.871745 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869250 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 15:56:40.871745 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869252 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 15:56:40.871745 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869254 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 15:56:40.871745 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869257 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 15:56:40.871745 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869260 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 15:56:40.871745 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869262 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 15:56:40.871745 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869264 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 15:56:40.871745 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869267 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 15:56:40.871745 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869269 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 15:56:40.871745 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869272 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 15:56:40.871745 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869274 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 15:56:40.871745 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869276 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 15:56:40.871745 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869278 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 15:56:40.871745 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869281 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 15:56:40.872215 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869283 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 15:56:40.872215 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869286 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 15:56:40.872215 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869288 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 15:56:40.872215 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869290 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 15:56:40.872215 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869292 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 15:56:40.872215 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869295 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 15:56:40.872215 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869297 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 15:56:40.872215 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869300 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 15:56:40.872215 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869303 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 15:56:40.872215 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869306 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 15:56:40.872215 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869309 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 15:56:40.872215 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869311 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 15:56:40.872215 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869314 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 15:56:40.872215 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.869318 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 15:56:40.872215 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869446 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 15:56:40.872624 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869450 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 15:56:40.872624 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869453 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 15:56:40.872624 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869455 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 15:56:40.872624 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869458 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 15:56:40.872624 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869460 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 15:56:40.872624 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869463 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 15:56:40.872624 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869472 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 15:56:40.872624 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869475 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 15:56:40.872624 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869478 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 15:56:40.872624 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869480 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 15:56:40.872624 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869482 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 15:56:40.872624 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869485 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 15:56:40.872624 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869488 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 15:56:40.872624 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869490 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 15:56:40.872624 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869492 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 15:56:40.872624 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869495 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 15:56:40.872624 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869497 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 15:56:40.872624 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869500 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 15:56:40.872624 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869502 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 15:56:40.872624 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869504 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 15:56:40.873065 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869508 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 15:56:40.873065 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869511 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 15:56:40.873065 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869514 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 15:56:40.873065 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869517 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 15:56:40.873065 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869520 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 15:56:40.873065 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869522 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 15:56:40.873065 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869525 2573 feature_gate.go:328] unrecognized feature gate: Example Aug 04 15:56:40.873065 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869529 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 15:56:40.873065 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869532 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 15:56:40.873065 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869534 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 15:56:40.873065 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869537 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 15:56:40.873065 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869539 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 15:56:40.873065 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869560 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 15:56:40.873065 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869564 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 15:56:40.873065 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869567 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 15:56:40.873065 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869570 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 15:56:40.873065 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869572 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 15:56:40.873065 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869575 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 15:56:40.873065 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869577 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 15:56:40.873513 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869580 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 15:56:40.873513 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869582 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 15:56:40.873513 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869584 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 15:56:40.873513 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869587 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 15:56:40.873513 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869589 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 15:56:40.873513 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869592 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 15:56:40.873513 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869594 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 15:56:40.873513 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869596 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 15:56:40.873513 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869599 2573 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 15:56:40.873513 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869602 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 15:56:40.873513 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869604 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 15:56:40.873513 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869607 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 15:56:40.873513 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869611 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 15:56:40.873513 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869614 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 15:56:40.873513 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869617 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 15:56:40.873513 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869619 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 15:56:40.873513 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869622 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 15:56:40.873513 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869624 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 15:56:40.873513 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869627 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 15:56:40.873513 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869630 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 15:56:40.873983 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869639 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 15:56:40.873983 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869643 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 15:56:40.873983 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869645 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 15:56:40.873983 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869648 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 15:56:40.873983 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869650 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 15:56:40.873983 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869653 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 15:56:40.873983 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869655 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 15:56:40.873983 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869657 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 15:56:40.873983 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869659 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 15:56:40.873983 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869662 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 15:56:40.873983 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869664 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 15:56:40.873983 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869666 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 15:56:40.873983 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869669 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 15:56:40.873983 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869671 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 15:56:40.873983 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869673 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 15:56:40.873983 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869676 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 15:56:40.873983 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869678 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 15:56:40.873983 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869680 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 15:56:40.873983 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869683 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 15:56:40.873983 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869685 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 15:56:40.874421 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869688 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 15:56:40.874421 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869690 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 15:56:40.874421 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869693 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 15:56:40.874421 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869695 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 15:56:40.874421 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869697 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 15:56:40.874421 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869700 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 15:56:40.874421 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869702 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 15:56:40.874421 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869704 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 15:56:40.874421 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869707 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 15:56:40.874421 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869709 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 15:56:40.874421 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869712 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 15:56:40.874421 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869714 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 15:56:40.874421 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869716 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 15:56:40.874421 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869719 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 15:56:40.874421 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869722 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 15:56:40.874421 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869725 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 15:56:40.874421 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869731 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 15:56:40.874421 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869733 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 15:56:40.874421 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:40.869736 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 15:56:40.874850 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.869740 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 15:56:40.874850 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.869858 2573 server.go:962] "Client rotation is on, will bootstrap in background" Aug 04 15:56:40.874850 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.872074 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Aug 04 15:56:40.874850 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.873426 2573 server.go:1019] "Starting client certificate rotation" Aug 04 15:56:40.874850 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.873523 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 15:56:40.874850 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.874176 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 15:56:40.876306 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.876294 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 15:56:40.878976 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.878960 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 15:56:40.889747 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.889724 2573 log.go:25] "Validated CRI v1 runtime API" Aug 04 15:56:40.893532 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.893507 2573 log.go:25] "Validated CRI v1 image API" Aug 04 15:56:40.894222 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.894207 2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Aug 04 15:56:40.898637 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.898617 2573 fs.go:135] Filesystem UUIDs: map[5b6456f0-0f5d-4f37-af00-6f5f0bbae799:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f9453067-0f90-41dd-a6cd-79bed548e285:/dev/nvme0n1p4] Aug 04 15:56:40.898701 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.898635 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Aug 04 15:56:40.902832 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.902814 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 15:56:40.904517 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.904408 2573 manager.go:216] Machine: {Timestamp:2026-08-04 15:56:40.903134867 +0000 UTC m=+0.320676981 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100458 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d78d33944f31d3e4bc108937299fe SystemUUID:ec2d78d3-3944-f31d-3e4b-c108937299fe BootID:535aff61-49b1-4c7a-965f-89d13589ee87 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:da:4e:e2:ef:1d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:da:4e:e2:ef:1d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ee:0d:e6:ce:46:8b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Aug 04 15:56:40.904517 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.904514 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Aug 04 15:56:40.904642 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.904605 2573 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Aug 04 15:56:40.906187 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.906168 2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Aug 04 15:56:40.906320 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.906189 2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-210.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Aug 04 15:56:40.906948 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.906939 2573 topology_manager.go:138] "Creating topology manager with none policy" Aug 04 15:56:40.906976 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.906951 2573 container_manager_linux.go:309] "Creating device plugin manager" Aug 04 15:56:40.906976 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.906958 2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 15:56:40.907660 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.907651 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 15:56:40.909290 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.909281 2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Aug 04 15:56:40.913346 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.913334 2573 state_mem.go:36] "Initialized new in-memory state store" Aug 04 15:56:40.913453 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.913445 2573 server.go:1240] "Using root directory" path="/var/lib/kubelet" Aug 04 15:56:40.913485 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.913475 2573 kubelet.go:486] "Attempting to sync node with API server" Aug 04 15:56:40.913510 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.913485 2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Aug 04 15:56:40.913510 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.913495 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Aug 04 15:56:40.913510 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.913503 2573 kubelet.go:398] "Adding apiserver pod source" Aug 04 15:56:40.913626 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.913512 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Aug 04 15:56:40.913873 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.913863 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 15:56:40.913903 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.913886 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 15:56:40.916072 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.916055 2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Aug 04 15:56:40.917756 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.917744 2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Aug 04 15:56:40.917804 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.917771 2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Aug 04 15:56:40.918369 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.918357 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Aug 04 15:56:40.918456 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.918374 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Aug 04 15:56:40.918456 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.918382 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Aug 04 15:56:40.918456 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.918389 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Aug 04 15:56:40.918456 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.918399 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Aug 04 15:56:40.918456 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.918404 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Aug 04 15:56:40.918661 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.918571 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Aug 04 15:56:40.918661 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.918577 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Aug 04 15:56:40.918661 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.918584 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Aug 04 15:56:40.918661 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.918589 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Aug 04 15:56:40.918661 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.918598 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Aug 04 15:56:40.918661 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.918606 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Aug 04 15:56:40.919356 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.919347 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Aug 04 15:56:40.919356 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.919354 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Aug 04 15:56:40.922566 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.922427 2573 server.go:1268] "Started kubelet" Aug 04 15:56:40.922566 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.922461 2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Aug 04 15:56:40.922686 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.922597 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Aug 04 15:56:40.922730 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.922688 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Aug 04 15:56:40.923446 ip-10-0-141-210 systemd[1]: Started Kubernetes Kubelet. Aug 04 15:56:40.924088 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.924037 2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Aug 04 15:56:40.925067 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:40.925028 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Aug 04 15:56:40.925157 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:40.925066 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-210.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Aug 04 15:56:40.925157 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.925078 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-210.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Aug 04 15:56:40.925400 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.925386 2573 server.go:310] "Adding debug handlers to kubelet server" Aug 04 15:56:40.928537 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:40.928516 2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Aug 04 15:56:40.931672 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:40.930509 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-210.ec2.internal.18c8a54ca8e6bd79 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-210.ec2.internal,UID:ip-10-0-141-210.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-210.ec2.internal,},FirstTimestamp:2026-08-04 15:56:40.922193273 +0000 UTC m=+0.339735387,LastTimestamp:2026-08-04 15:56:40.922193273 +0000 UTC m=+0.339735387,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-210.ec2.internal,}" Aug 04 15:56:40.932213 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.932195 2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Aug 04 15:56:40.932213 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.932205 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Aug 04 15:56:40.932775 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:40.932370 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-210.ec2.internal\" not found" Aug 04 15:56:40.932876 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.932842 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Aug 04 15:56:40.932876 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.932842 2573 volume_manager.go:311] "The desired_state_of_world populator starts" Aug 04 15:56:40.932876 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.932870 2573 volume_manager.go:313] "Starting Kubelet Volume Manager" Aug 04 15:56:40.933014 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.932984 2573 factory.go:55] Registering systemd factory Aug 04 15:56:40.933014 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.933000 2573 factory.go:223] Registration of the systemd container factory successfully Aug 04 15:56:40.933014 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.932986 2573 reconstruct.go:97] "Volume reconstruction finished" Aug 04 15:56:40.933138 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.933022 2573 reconciler.go:29] "Reconciler: start to sync state" Aug 04 15:56:40.933221 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.933210 2573 factory.go:153] Registering CRI-O factory Aug 04 15:56:40.933252 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.933225 2573 factory.go:223] Registration of the crio container factory successfully Aug 04 15:56:40.933287 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.933276 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Aug 04 15:56:40.933320 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.933298 2573 factory.go:103] Registering Raw factory Aug 04 15:56:40.933320 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.933313 2573 manager.go:1196] Started watching for new ooms in manager Aug 04 15:56:40.933917 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.933895 2573 manager.go:349] Starting recovery of all containers Aug 04 15:56:40.934971 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.934947 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5gxm2" Aug 04 15:56:40.936269 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:40.936247 2573 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-210.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Aug 04 15:56:40.936269 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:40.936251 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Aug 04 15:56:40.942383 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.942367 2573 manager.go:354] Recovery completed Aug 04 15:56:40.946270 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.946241 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5gxm2" Aug 04 15:56:40.947693 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.947682 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 15:56:40.949403 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.949387 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-210.ec2.internal" event="NodeHasSufficientMemory" Aug 04 15:56:40.949451 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.949420 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-210.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 15:56:40.949451 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.949431 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-210.ec2.internal" event="NodeHasSufficientPID" Aug 04 15:56:40.949912 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.949900 2573 cpu_manager.go:222] "Starting CPU manager" policy="none" Aug 04 15:56:40.949912 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.949910 2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Aug 04 15:56:40.949999 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.949925 2573 state_mem.go:36] "Initialized new in-memory state store" Aug 04 15:56:40.949999 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.949908 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 15:56:40.950764 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.950672 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-210.ec2.internal" event="NodeHasSufficientMemory" Aug 04 15:56:40.950764 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.950703 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-210.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 15:56:40.950764 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.950717 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-210.ec2.internal" event="NodeHasSufficientPID" Aug 04 15:56:40.950900 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.950885 2573 policy_none.go:49] "None policy: Start" Aug 04 15:56:40.950941 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.950903 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Aug 04 15:56:40.950941 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.950914 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Aug 04 15:56:40.951739 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:40.951678 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-210.ec2.internal.18c8a54caa85f75b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-210.ec2.internal,UID:ip-10-0-141-210.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-210.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-210.ec2.internal,},FirstTimestamp:2026-08-04 15:56:40.949405531 +0000 UTC m=+0.366947656,LastTimestamp:2026-08-04 15:56:40.949405531 +0000 UTC m=+0.366947656,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-210.ec2.internal,}" Aug 04 15:56:40.951824 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:40.951769 2573 policy_none.go:47] "Start" Aug 04 15:56:41.000354 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.000203 2573 manager.go:337] "Starting Device Plugin manager" Aug 04 15:56:41.000468 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:41.000370 2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Aug 04 15:56:41.000468 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.000380 2573 server.go:87] "Starting device plugin registration server" Aug 04 15:56:41.000704 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.000692 2573 eviction_manager.go:189] "Eviction manager: starting control loop" Aug 04 15:56:41.000741 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.000708 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Aug 04 15:56:41.000822 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.000801 2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Aug 04 15:56:41.000913 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.000892 2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Aug 04 15:56:41.000913 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.000901 2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Aug 04 15:56:41.001347 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:41.001327 2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Aug 04 15:56:41.001425 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:41.001373 2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-210.ec2.internal\" not found" Aug 04 15:56:41.071161 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.071089 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Aug 04 15:56:41.072257 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.072241 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Aug 04 15:56:41.072319 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.072278 2573 status_manager.go:244] "Starting to sync pod status with apiserver" Aug 04 15:56:41.072319 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.072302 2573 kubelet.go:2448] "Starting kubelet main sync loop" Aug 04 15:56:41.072384 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:41.072336 2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Aug 04 15:56:41.076121 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.076103 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 15:56:41.101209 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.101178 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 15:56:41.102084 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.102065 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-210.ec2.internal" event="NodeHasSufficientMemory" Aug 04 15:56:41.102180 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.102098 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-210.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 15:56:41.102180 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.102108 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-210.ec2.internal" event="NodeHasSufficientPID" Aug 04 15:56:41.102180 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.102129 2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-210.ec2.internal" Aug 04 15:56:41.111920 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.111902 2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-210.ec2.internal" Aug 04 15:56:41.111971 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:41.111923 2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-210.ec2.internal\": node \"ip-10-0-141-210.ec2.internal\" not found" Aug 04 15:56:41.142418 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:41.142393 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-210.ec2.internal\" not found" Aug 04 15:56:41.173168 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.173133 2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-210.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-210.ec2.internal"] Aug 04 15:56:41.173341 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.173227 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 15:56:41.174453 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.174437 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-210.ec2.internal" event="NodeHasSufficientMemory" Aug 04 15:56:41.174580 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.174469 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-210.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 15:56:41.174580 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.174484 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-210.ec2.internal" event="NodeHasSufficientPID" Aug 04 15:56:41.176097 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.176082 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 15:56:41.176190 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.176176 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-210.ec2.internal" Aug 04 15:56:41.176239 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.176226 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 15:56:41.176829 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.176810 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-210.ec2.internal" event="NodeHasSufficientMemory" Aug 04 15:56:41.176909 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.176836 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-210.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 15:56:41.176909 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.176849 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-210.ec2.internal" event="NodeHasSufficientPID" Aug 04 15:56:41.176909 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.176849 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-210.ec2.internal" event="NodeHasSufficientMemory" Aug 04 15:56:41.177011 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.176914 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-210.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 15:56:41.177011 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.176925 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-210.ec2.internal" event="NodeHasSufficientPID" Aug 04 15:56:41.178458 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.178444 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-210.ec2.internal" Aug 04 15:56:41.178537 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.178468 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 15:56:41.179089 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.179076 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-210.ec2.internal" event="NodeHasSufficientMemory" Aug 04 15:56:41.179144 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.179098 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-210.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 15:56:41.179144 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.179107 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-210.ec2.internal" event="NodeHasSufficientPID" Aug 04 15:56:41.204129 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:41.204101 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-210.ec2.internal\" not found" node="ip-10-0-141-210.ec2.internal" Aug 04 15:56:41.208157 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:41.208140 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-210.ec2.internal\" not found" node="ip-10-0-141-210.ec2.internal" Aug 04 15:56:41.234150 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.234120 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e35ae5863cdf97bdb5252257e0c413f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-210.ec2.internal\" (UID: \"5e35ae5863cdf97bdb5252257e0c413f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-210.ec2.internal" Aug 04 15:56:41.234150 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.234149 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fd26de1bd1ac6d99210ec0097a02c100-config\") pod \"kube-apiserver-proxy-ip-10-0-141-210.ec2.internal\" (UID: \"fd26de1bd1ac6d99210ec0097a02c100\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-210.ec2.internal" Aug 04 15:56:41.234295 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.234165 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5e35ae5863cdf97bdb5252257e0c413f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-210.ec2.internal\" (UID: \"5e35ae5863cdf97bdb5252257e0c413f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-210.ec2.internal" Aug 04 15:56:41.243043 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:41.243021 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-210.ec2.internal\" not found" Aug 04 15:56:41.334917 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.334837 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5e35ae5863cdf97bdb5252257e0c413f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-210.ec2.internal\" (UID: \"5e35ae5863cdf97bdb5252257e0c413f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-210.ec2.internal" Aug 04 15:56:41.334917 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.334867 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e35ae5863cdf97bdb5252257e0c413f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-210.ec2.internal\" (UID: \"5e35ae5863cdf97bdb5252257e0c413f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-210.ec2.internal" Aug 04 15:56:41.334917 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.334887 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fd26de1bd1ac6d99210ec0097a02c100-config\") pod \"kube-apiserver-proxy-ip-10-0-141-210.ec2.internal\" (UID: \"fd26de1bd1ac6d99210ec0097a02c100\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-210.ec2.internal" Aug 04 15:56:41.335108 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.334937 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fd26de1bd1ac6d99210ec0097a02c100-config\") pod \"kube-apiserver-proxy-ip-10-0-141-210.ec2.internal\" (UID: \"fd26de1bd1ac6d99210ec0097a02c100\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-210.ec2.internal" Aug 04 15:56:41.335108 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.334953 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e35ae5863cdf97bdb5252257e0c413f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-210.ec2.internal\" (UID: \"5e35ae5863cdf97bdb5252257e0c413f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-210.ec2.internal" Aug 04 15:56:41.335108 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.334937 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5e35ae5863cdf97bdb5252257e0c413f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-210.ec2.internal\" (UID: \"5e35ae5863cdf97bdb5252257e0c413f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-210.ec2.internal" Aug 04 15:56:41.343638 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:41.343614 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-210.ec2.internal\" not found" Aug 04 15:56:41.444611 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:41.444584 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-210.ec2.internal\" not found" Aug 04 15:56:41.506751 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.506715 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-210.ec2.internal" Aug 04 15:56:41.511900 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.511883 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-210.ec2.internal" Aug 04 15:56:41.545486 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:41.545455 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-210.ec2.internal\" not found" Aug 04 15:56:41.645975 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:41.645897 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-210.ec2.internal\" not found" Aug 04 15:56:41.746281 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:41.746248 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-210.ec2.internal\" not found" Aug 04 15:56:41.846700 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:41.846663 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-210.ec2.internal\" not found" Aug 04 15:56:41.872812 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.872787 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Aug 04 15:56:41.873254 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.872946 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 04 15:56:41.912840 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.912775 2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 15:56:41.947361 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:41.947331 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-210.ec2.internal\" not found" Aug 04 15:56:41.950434 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.950396 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-03 15:51:40 +0000 UTC" deadline="2028-02-18 11:31:35.15783518 +0000 UTC" Aug 04 15:56:41.950519 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.950435 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13507h34m53.207403548s" Aug 04 15:56:41.984222 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:41.984187 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e35ae5863cdf97bdb5252257e0c413f.slice/crio-5bdbfaf84601e5d90f7e35acdd1142e6a75f6286a6edcf5ea83c1a69b7f3f6a8 WatchSource:0}: Error finding container 5bdbfaf84601e5d90f7e35acdd1142e6a75f6286a6edcf5ea83c1a69b7f3f6a8: Status 404 returned error can't find the container with id 5bdbfaf84601e5d90f7e35acdd1142e6a75f6286a6edcf5ea83c1a69b7f3f6a8 Aug 04 15:56:41.987928 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:41.987916 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 15:56:42.048413 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:42.048375 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-210.ec2.internal\" not found" Aug 04 15:56:42.075450 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.075407 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-210.ec2.internal" event={"ID":"5e35ae5863cdf97bdb5252257e0c413f","Type":"ContainerStarted","Data":"5bdbfaf84601e5d90f7e35acdd1142e6a75f6286a6edcf5ea83c1a69b7f3f6a8"} Aug 04 15:56:42.148818 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:42.148784 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-210.ec2.internal\" not found" Aug 04 15:56:42.205184 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.205126 2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 15:56:42.232783 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.232751 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-210.ec2.internal" Aug 04 15:56:42.246011 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.245990 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 15:56:42.246992 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.246975 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-210.ec2.internal" Aug 04 15:56:42.256714 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.256689 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 15:56:42.350019 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.349994 2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 15:56:42.900755 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.900729 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 15:56:42.915283 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.915258 2573 apiserver.go:52] "Watching apiserver" Aug 04 15:56:42.923606 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.923573 2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Aug 04 15:56:42.923928 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.923906 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-q2zlj","kube-system/konnectivity-agent-8jnqc","kube-system/kube-apiserver-proxy-ip-10-0-141-210.ec2.internal","openshift-dns/node-resolver-zfqf9","openshift-network-diagnostics/network-check-target-bc6jp","openshift-ovn-kubernetes/ovnkube-node-mtbc5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lccb2","openshift-cluster-node-tuning-operator/tuned-zcx94","openshift-image-registry/node-ca-rz25w","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-210.ec2.internal","openshift-multus/multus-additional-cni-plugins-bgqfs","openshift-multus/multus-jbkjx","openshift-multus/network-metrics-daemon-rcb4z","openshift-network-operator/iptables-alerter-kg84j"] Aug 04 15:56:42.925411 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.925389 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lccb2" Aug 04 15:56:42.926734 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.926711 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bgqfs" Aug 04 15:56:42.927430 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.927406 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Aug 04 15:56:42.927637 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.927411 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Aug 04 15:56:42.927637 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.927456 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-m5r5t\"" Aug 04 15:56:42.927637 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.927517 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Aug 04 15:56:42.927797 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.927740 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kg84j" Aug 04 15:56:42.928729 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.928711 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Aug 04 15:56:42.928943 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.928924 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zfqf9" Aug 04 15:56:42.929237 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.928930 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Aug 04 15:56:42.929708 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.929530 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Aug 04 15:56:42.929708 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.929596 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Aug 04 15:56:42.929708 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.929606 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Aug 04 15:56:42.929708 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.929531 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5gkdk\"" Aug 04 15:56:42.929708 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.929701 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Aug 04 15:56:42.929958 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.929832 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Aug 04 15:56:42.929958 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.929922 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pcxr4\"" Aug 04 15:56:42.930048 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.930016 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Aug 04 15:56:42.930048 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.930037 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc6jp" Aug 04 15:56:42.930127 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:42.930103 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bc6jp" podUID="cd2d6392-5d3a-49f6-be6a-954d6aaffba6" Aug 04 15:56:42.930476 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.930456 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zvgfv\"" Aug 04 15:56:42.930613 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.930490 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Aug 04 15:56:42.930760 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.930738 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Aug 04 15:56:42.931559 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.931466 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:42.932596 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.932577 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Aug 04 15:56:42.932689 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.932679 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rz25w" Aug 04 15:56:42.933273 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.933256 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Aug 04 15:56:42.933348 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.933302 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Aug 04 15:56:42.933897 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.933714 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Aug 04 15:56:42.933897 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.933725 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:42.933897 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.933714 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6hmmr\"" Aug 04 15:56:42.933897 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.933813 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Aug 04 15:56:42.934679 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.934663 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Aug 04 15:56:42.935145 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.935126 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8jnqc" Aug 04 15:56:42.935497 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.935478 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Aug 04 15:56:42.935605 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.935592 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Aug 04 15:56:42.935806 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.935786 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Aug 04 15:56:42.935905 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.935814 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Aug 04 15:56:42.935905 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.935888 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Aug 04 15:56:42.935996 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.935944 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rxqrm\"" Aug 04 15:56:42.936084 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.936046 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s2tnd\"" Aug 04 15:56:42.936186 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.936168 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Aug 04 15:56:42.936505 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.936489 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jbkjx" Aug 04 15:56:42.936836 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.936819 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2lqcl\"" Aug 04 15:56:42.937387 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.937369 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Aug 04 15:56:42.937498 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.937445 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Aug 04 15:56:42.937807 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.937792 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zlj" Aug 04 15:56:42.937872 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:42.937852 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2zlj" podUID="e05ec517-319e-4360-8a0f-6133fb238192" Aug 04 15:56:42.938278 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.938260 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Aug 04 15:56:42.938360 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.938346 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ftp6n\"" Aug 04 15:56:42.938992 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.938979 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcb4z" Aug 04 15:56:42.939075 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:42.939044 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcb4z" podUID="fb454294-c3a4-4d8d-b705-95350ed95c13" Aug 04 15:56:42.941140 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.941120 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28b7m\" (UniqueName: \"kubernetes.io/projected/11f9cbee-a773-420b-9873-5c8cb7679dd8-kube-api-access-28b7m\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:42.941265 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.941249 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:42.941396 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.941381 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-etc-kubernetes\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:42.941513 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.941499 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-host\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:42.941651 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.941637 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/588d2847-8d2d-4f62-97f0-2253584d2e07-etc-selinux\") pod \"aws-ebs-csi-driver-node-lccb2\" (UID: \"588d2847-8d2d-4f62-97f0-2253584d2e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lccb2" Aug 04 15:56:42.942047 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.941738 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67933315-d76a-4b2c-8c92-f0249e607072-cnibin\") pod \"multus-additional-cni-plugins-bgqfs\" (UID: \"67933315-d76a-4b2c-8c92-f0249e607072\") " pod="openshift-multus/multus-additional-cni-plugins-bgqfs" Aug 04 15:56:42.942047 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.941769 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8b28\" (UniqueName: \"kubernetes.io/projected/67933315-d76a-4b2c-8c92-f0249e607072-kube-api-access-q8b28\") pod \"multus-additional-cni-plugins-bgqfs\" (UID: \"67933315-d76a-4b2c-8c92-f0249e607072\") " pod="openshift-multus/multus-additional-cni-plugins-bgqfs" Aug 04 15:56:42.942047 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.941798 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/11f9cbee-a773-420b-9873-5c8cb7679dd8-ovn-node-metrics-cert\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:42.942047 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.941826 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-run\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:42.942047 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.941868 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzw72\" (UniqueName: \"kubernetes.io/projected/482db4e4-6c5f-4c1e-9666-88f470fe67db-kube-api-access-pzw72\") pod \"node-resolver-zfqf9\" (UID: \"482db4e4-6c5f-4c1e-9666-88f470fe67db\") " pod="openshift-dns/node-resolver-zfqf9" Aug 04 15:56:42.942047 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.941905 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/482db4e4-6c5f-4c1e-9666-88f470fe67db-hosts-file\") pod \"node-resolver-zfqf9\" (UID: \"482db4e4-6c5f-4c1e-9666-88f470fe67db\") " pod="openshift-dns/node-resolver-zfqf9" Aug 04 15:56:42.942047 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.941941 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bmm5\" (UniqueName: \"kubernetes.io/projected/cd2d6392-5d3a-49f6-be6a-954d6aaffba6-kube-api-access-6bmm5\") pod \"network-check-target-bc6jp\" (UID: \"cd2d6392-5d3a-49f6-be6a-954d6aaffba6\") " pod="openshift-network-diagnostics/network-check-target-bc6jp" Aug 04 15:56:42.942047 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.941966 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c10e0ff5-2215-4032-976a-907e6dccb2d3-host\") pod \"node-ca-rz25w\" (UID: \"c10e0ff5-2215-4032-976a-907e6dccb2d3\") " pod="openshift-image-registry/node-ca-rz25w" Aug 04 15:56:42.942047 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.941992 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-etc-sysconfig\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:42.942047 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.942013 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-var-lib-kubelet\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:42.942478 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.942060 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-node-log\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:42.942478 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.942093 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-host-run-ovn-kubernetes\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:42.942478 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.942120 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/11f9cbee-a773-420b-9873-5c8cb7679dd8-ovnkube-script-lib\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:42.942478 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.942164 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-host-cni-netd\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:42.942478 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.942190 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/482db4e4-6c5f-4c1e-9666-88f470fe67db-tmp-dir\") pod \"node-resolver-zfqf9\" (UID: \"482db4e4-6c5f-4c1e-9666-88f470fe67db\") " pod="openshift-dns/node-resolver-zfqf9" Aug 04 15:56:42.942478 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.942210 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/588d2847-8d2d-4f62-97f0-2253584d2e07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lccb2\" (UID: \"588d2847-8d2d-4f62-97f0-2253584d2e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lccb2" Aug 04 15:56:42.942478 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.942236 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qxb4\" (UniqueName: \"kubernetes.io/projected/3932e5e3-69c2-4cee-ad2b-ac496485b140-kube-api-access-7qxb4\") pod \"iptables-alerter-kg84j\" (UID: \"3932e5e3-69c2-4cee-ad2b-ac496485b140\") " pod="openshift-network-operator/iptables-alerter-kg84j" Aug 04 15:56:42.942478 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.942285 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-etc-sysctl-d\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:42.942478 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.942311 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-etc-tuned\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:42.942478 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.942332 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/588d2847-8d2d-4f62-97f0-2253584d2e07-sys-fs\") pod \"aws-ebs-csi-driver-node-lccb2\" (UID: \"588d2847-8d2d-4f62-97f0-2253584d2e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lccb2" Aug 04 15:56:42.942478 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.942356 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-etc-openvswitch\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:42.942478 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.942380 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/67933315-d76a-4b2c-8c92-f0249e607072-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bgqfs\" (UID: \"67933315-d76a-4b2c-8c92-f0249e607072\") " pod="openshift-multus/multus-additional-cni-plugins-bgqfs" Aug 04 15:56:42.942478 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.942409 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/67933315-d76a-4b2c-8c92-f0249e607072-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bgqfs\" (UID: \"67933315-d76a-4b2c-8c92-f0249e607072\") " pod="openshift-multus/multus-additional-cni-plugins-bgqfs" Aug 04 15:56:42.942478 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.942435 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3932e5e3-69c2-4cee-ad2b-ac496485b140-iptables-alerter-script\") pod \"iptables-alerter-kg84j\" (UID: \"3932e5e3-69c2-4cee-ad2b-ac496485b140\") " pod="openshift-network-operator/iptables-alerter-kg84j" Aug 04 15:56:42.943114 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.942479 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3932e5e3-69c2-4cee-ad2b-ac496485b140-host-slash\") pod \"iptables-alerter-kg84j\" (UID: \"3932e5e3-69c2-4cee-ad2b-ac496485b140\") " pod="openshift-network-operator/iptables-alerter-kg84j" Aug 04 15:56:42.943114 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.942565 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-host-kubelet\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:42.943114 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.942595 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-systemd-units\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:42.943114 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.942622 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/11f9cbee-a773-420b-9873-5c8cb7679dd8-ovnkube-config\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:42.943114 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.942642 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-etc-modprobe-d\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:42.943114 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.942680 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-log-socket\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:42.943114 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.942706 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-host-cni-bin\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:42.943114 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.942733 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-sys\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:42.943114 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.942752 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-tmp\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:42.943114 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.942778 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/67933315-d76a-4b2c-8c92-f0249e607072-os-release\") pod \"multus-additional-cni-plugins-bgqfs\" (UID: \"67933315-d76a-4b2c-8c92-f0249e607072\") " pod="openshift-multus/multus-additional-cni-plugins-bgqfs" Aug 04 15:56:42.943114 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.942807 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67933315-d76a-4b2c-8c92-f0249e607072-cni-binary-copy\") pod \"multus-additional-cni-plugins-bgqfs\" (UID: \"67933315-d76a-4b2c-8c92-f0249e607072\") " pod="openshift-multus/multus-additional-cni-plugins-bgqfs" Aug 04 15:56:42.943114 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.942833 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-run-systemd\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:42.943114 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.942858 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-run-openvswitch\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:42.943114 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.942881 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/11f9cbee-a773-420b-9873-5c8cb7679dd8-env-overrides\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:42.943114 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.942910 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/588d2847-8d2d-4f62-97f0-2253584d2e07-registration-dir\") pod \"aws-ebs-csi-driver-node-lccb2\" (UID: \"588d2847-8d2d-4f62-97f0-2253584d2e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lccb2" Aug 04 15:56:42.943114 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.942936 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c10e0ff5-2215-4032-976a-907e6dccb2d3-serviceca\") pod \"node-ca-rz25w\" (UID: \"c10e0ff5-2215-4032-976a-907e6dccb2d3\") " pod="openshift-image-registry/node-ca-rz25w" Aug 04 15:56:42.943796 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.943028 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-etc-sysctl-conf\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:42.943796 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.943054 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-lib-modules\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:42.943796 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.943085 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj5mb\" (UniqueName: \"kubernetes.io/projected/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-kube-api-access-gj5mb\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:42.943796 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.943114 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67933315-d76a-4b2c-8c92-f0249e607072-system-cni-dir\") pod \"multus-additional-cni-plugins-bgqfs\" (UID: \"67933315-d76a-4b2c-8c92-f0249e607072\") " pod="openshift-multus/multus-additional-cni-plugins-bgqfs" Aug 04 15:56:42.943796 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.943144 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-host-run-netns\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:42.943796 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.943248 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/588d2847-8d2d-4f62-97f0-2253584d2e07-socket-dir\") pod \"aws-ebs-csi-driver-node-lccb2\" (UID: \"588d2847-8d2d-4f62-97f0-2253584d2e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lccb2" Aug 04 15:56:42.943796 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.943313 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67933315-d76a-4b2c-8c92-f0249e607072-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bgqfs\" (UID: \"67933315-d76a-4b2c-8c92-f0249e607072\") " pod="openshift-multus/multus-additional-cni-plugins-bgqfs" Aug 04 15:56:42.943796 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.943403 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkjf5\" (UniqueName: \"kubernetes.io/projected/c10e0ff5-2215-4032-976a-907e6dccb2d3-kube-api-access-pkjf5\") pod \"node-ca-rz25w\" (UID: \"c10e0ff5-2215-4032-976a-907e6dccb2d3\") " pod="openshift-image-registry/node-ca-rz25w" Aug 04 15:56:42.943796 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.943438 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-etc-systemd\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:42.943796 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.943491 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/588d2847-8d2d-4f62-97f0-2253584d2e07-device-dir\") pod \"aws-ebs-csi-driver-node-lccb2\" (UID: \"588d2847-8d2d-4f62-97f0-2253584d2e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lccb2" Aug 04 15:56:42.943796 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.943518 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pq7k\" (UniqueName: \"kubernetes.io/projected/588d2847-8d2d-4f62-97f0-2253584d2e07-kube-api-access-2pq7k\") pod \"aws-ebs-csi-driver-node-lccb2\" (UID: \"588d2847-8d2d-4f62-97f0-2253584d2e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lccb2" Aug 04 15:56:42.943796 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.943581 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-host-slash\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:42.943796 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.943627 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-var-lib-openvswitch\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:42.943796 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.943683 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-run-ovn\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:42.945431 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.945412 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 15:56:42.977682 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.977660 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rgkt7" Aug 04 15:56:42.986935 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:42.986914 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rgkt7" Aug 04 15:56:43.034530 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.034498 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Aug 04 15:56:43.043889 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.043852 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp5cx\" (UniqueName: \"kubernetes.io/projected/fb454294-c3a4-4d8d-b705-95350ed95c13-kube-api-access-lp5cx\") pod \"network-metrics-daemon-rcb4z\" (UID: \"fb454294-c3a4-4d8d-b705-95350ed95c13\") " pod="openshift-multus/network-metrics-daemon-rcb4z" Aug 04 15:56:43.043889 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.043893 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-host-var-lib-kubelet\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.044120 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.043918 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/abba07c5-995b-4a73-8a11-cde3a8326227-multus-daemon-config\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.044120 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.043948 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/67933315-d76a-4b2c-8c92-f0249e607072-os-release\") pod \"multus-additional-cni-plugins-bgqfs\" (UID: \"67933315-d76a-4b2c-8c92-f0249e607072\") " pod="openshift-multus/multus-additional-cni-plugins-bgqfs" Aug 04 15:56:43.044120 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044032 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/67933315-d76a-4b2c-8c92-f0249e607072-os-release\") pod \"multus-additional-cni-plugins-bgqfs\" (UID: \"67933315-d76a-4b2c-8c92-f0249e607072\") " pod="openshift-multus/multus-additional-cni-plugins-bgqfs" Aug 04 15:56:43.044120 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044032 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67933315-d76a-4b2c-8c92-f0249e607072-cni-binary-copy\") pod \"multus-additional-cni-plugins-bgqfs\" (UID: \"67933315-d76a-4b2c-8c92-f0249e607072\") " pod="openshift-multus/multus-additional-cni-plugins-bgqfs" Aug 04 15:56:43.044120 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044090 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-run-systemd\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.044325 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044148 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-run-openvswitch\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.044325 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044179 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/11f9cbee-a773-420b-9873-5c8cb7679dd8-env-overrides\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.044325 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044223 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-hostroot\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.044325 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044248 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/588d2847-8d2d-4f62-97f0-2253584d2e07-registration-dir\") pod \"aws-ebs-csi-driver-node-lccb2\" (UID: \"588d2847-8d2d-4f62-97f0-2253584d2e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lccb2" Aug 04 15:56:43.044325 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044272 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c10e0ff5-2215-4032-976a-907e6dccb2d3-serviceca\") pod \"node-ca-rz25w\" (UID: \"c10e0ff5-2215-4032-976a-907e6dccb2d3\") " pod="openshift-image-registry/node-ca-rz25w" Aug 04 15:56:43.044325 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044295 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-run-systemd\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.044325 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044319 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-etc-sysctl-conf\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:43.044652 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044362 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/588d2847-8d2d-4f62-97f0-2253584d2e07-registration-dir\") pod \"aws-ebs-csi-driver-node-lccb2\" (UID: \"588d2847-8d2d-4f62-97f0-2253584d2e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lccb2" Aug 04 15:56:43.044652 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044363 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-lib-modules\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:43.044652 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044403 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-run-openvswitch\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.044652 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044410 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj5mb\" (UniqueName: \"kubernetes.io/projected/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-kube-api-access-gj5mb\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:43.044652 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044449 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-host-run-netns\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.044652 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044476 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67933315-d76a-4b2c-8c92-f0249e607072-system-cni-dir\") pod \"multus-additional-cni-plugins-bgqfs\" (UID: \"67933315-d76a-4b2c-8c92-f0249e607072\") " pod="openshift-multus/multus-additional-cni-plugins-bgqfs" Aug 04 15:56:43.044652 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044498 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-host-run-netns\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.044652 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044521 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/975dcd50-33ff-4dbe-b7a8-43e44e042323-agent-certs\") pod \"konnectivity-agent-8jnqc\" (UID: \"975dcd50-33ff-4dbe-b7a8-43e44e042323\") " pod="kube-system/konnectivity-agent-8jnqc" Aug 04 15:56:43.044652 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044617 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e05ec517-319e-4360-8a0f-6133fb238192-dbus\") pod \"global-pull-secret-syncer-q2zlj\" (UID: \"e05ec517-319e-4360-8a0f-6133fb238192\") " pod="kube-system/global-pull-secret-syncer-q2zlj" Aug 04 15:56:43.044652 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044646 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-etc-sysctl-conf\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:43.045079 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044671 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67933315-d76a-4b2c-8c92-f0249e607072-cni-binary-copy\") pod \"multus-additional-cni-plugins-bgqfs\" (UID: \"67933315-d76a-4b2c-8c92-f0249e607072\") " pod="openshift-multus/multus-additional-cni-plugins-bgqfs" Aug 04 15:56:43.045079 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044674 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67933315-d76a-4b2c-8c92-f0249e607072-system-cni-dir\") pod \"multus-additional-cni-plugins-bgqfs\" (UID: \"67933315-d76a-4b2c-8c92-f0249e607072\") " pod="openshift-multus/multus-additional-cni-plugins-bgqfs" Aug 04 15:56:43.045079 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044687 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c10e0ff5-2215-4032-976a-907e6dccb2d3-serviceca\") pod \"node-ca-rz25w\" (UID: \"c10e0ff5-2215-4032-976a-907e6dccb2d3\") " pod="openshift-image-registry/node-ca-rz25w" Aug 04 15:56:43.045079 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044672 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-host-run-netns\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.045079 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044716 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-multus-cni-dir\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.045079 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044751 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-cnibin\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.045079 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044778 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/588d2847-8d2d-4f62-97f0-2253584d2e07-socket-dir\") pod \"aws-ebs-csi-driver-node-lccb2\" (UID: \"588d2847-8d2d-4f62-97f0-2253584d2e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lccb2" Aug 04 15:56:43.045079 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044798 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67933315-d76a-4b2c-8c92-f0249e607072-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bgqfs\" (UID: \"67933315-d76a-4b2c-8c92-f0249e607072\") " pod="openshift-multus/multus-additional-cni-plugins-bgqfs" Aug 04 15:56:43.045079 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044816 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-lib-modules\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:43.045079 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044821 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkjf5\" (UniqueName: \"kubernetes.io/projected/c10e0ff5-2215-4032-976a-907e6dccb2d3-kube-api-access-pkjf5\") pod \"node-ca-rz25w\" (UID: \"c10e0ff5-2215-4032-976a-907e6dccb2d3\") " pod="openshift-image-registry/node-ca-rz25w" Aug 04 15:56:43.045079 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044843 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-etc-systemd\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:43.045079 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044864 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/abba07c5-995b-4a73-8a11-cde3a8326227-cni-binary-copy\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.045079 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044884 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/588d2847-8d2d-4f62-97f0-2253584d2e07-device-dir\") pod \"aws-ebs-csi-driver-node-lccb2\" (UID: \"588d2847-8d2d-4f62-97f0-2253584d2e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lccb2" Aug 04 15:56:43.045079 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044889 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/588d2847-8d2d-4f62-97f0-2253584d2e07-socket-dir\") pod \"aws-ebs-csi-driver-node-lccb2\" (UID: \"588d2847-8d2d-4f62-97f0-2253584d2e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lccb2" Aug 04 15:56:43.045079 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044912 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pq7k\" (UniqueName: \"kubernetes.io/projected/588d2847-8d2d-4f62-97f0-2253584d2e07-kube-api-access-2pq7k\") pod \"aws-ebs-csi-driver-node-lccb2\" (UID: \"588d2847-8d2d-4f62-97f0-2253584d2e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lccb2" Aug 04 15:56:43.045079 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044916 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-etc-systemd\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:43.045079 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044941 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-host-slash\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.045805 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044949 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/588d2847-8d2d-4f62-97f0-2253584d2e07-device-dir\") pod \"aws-ebs-csi-driver-node-lccb2\" (UID: \"588d2847-8d2d-4f62-97f0-2253584d2e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lccb2" Aug 04 15:56:43.045805 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044950 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67933315-d76a-4b2c-8c92-f0249e607072-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bgqfs\" (UID: \"67933315-d76a-4b2c-8c92-f0249e607072\") " pod="openshift-multus/multus-additional-cni-plugins-bgqfs" Aug 04 15:56:43.045805 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044967 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-var-lib-openvswitch\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.045805 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044994 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-run-ovn\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.045805 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.044998 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-host-slash\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.045805 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045023 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28b7m\" (UniqueName: \"kubernetes.io/projected/11f9cbee-a773-420b-9873-5c8cb7679dd8-kube-api-access-28b7m\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.045805 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045050 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-multus-socket-dir-parent\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.045805 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045056 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-run-ovn\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.045805 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045023 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-var-lib-openvswitch\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.045805 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045076 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-host-run-multus-certs\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.045805 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045121 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.045805 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045162 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.045805 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045192 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-etc-kubernetes\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:43.045805 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045220 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-host\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:43.045805 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045244 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-os-release\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.045805 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045275 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/588d2847-8d2d-4f62-97f0-2253584d2e07-etc-selinux\") pod \"aws-ebs-csi-driver-node-lccb2\" (UID: \"588d2847-8d2d-4f62-97f0-2253584d2e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lccb2" Aug 04 15:56:43.045805 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045283 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-host\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:43.046503 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045290 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-etc-kubernetes\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:43.046503 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045298 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e05ec517-319e-4360-8a0f-6133fb238192-kubelet-config\") pod \"global-pull-secret-syncer-q2zlj\" (UID: \"e05ec517-319e-4360-8a0f-6133fb238192\") " pod="kube-system/global-pull-secret-syncer-q2zlj" Aug 04 15:56:43.046503 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045320 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb454294-c3a4-4d8d-b705-95350ed95c13-metrics-certs\") pod \"network-metrics-daemon-rcb4z\" (UID: \"fb454294-c3a4-4d8d-b705-95350ed95c13\") " pod="openshift-multus/network-metrics-daemon-rcb4z" Aug 04 15:56:43.046503 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045338 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-host-run-k8s-cni-cncf-io\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.046503 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045359 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67933315-d76a-4b2c-8c92-f0249e607072-cnibin\") pod \"multus-additional-cni-plugins-bgqfs\" (UID: \"67933315-d76a-4b2c-8c92-f0249e607072\") " pod="openshift-multus/multus-additional-cni-plugins-bgqfs" Aug 04 15:56:43.046503 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045378 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/588d2847-8d2d-4f62-97f0-2253584d2e07-etc-selinux\") pod \"aws-ebs-csi-driver-node-lccb2\" (UID: \"588d2847-8d2d-4f62-97f0-2253584d2e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lccb2" Aug 04 15:56:43.046503 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045403 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8b28\" (UniqueName: \"kubernetes.io/projected/67933315-d76a-4b2c-8c92-f0249e607072-kube-api-access-q8b28\") pod \"multus-additional-cni-plugins-bgqfs\" (UID: \"67933315-d76a-4b2c-8c92-f0249e607072\") " pod="openshift-multus/multus-additional-cni-plugins-bgqfs" Aug 04 15:56:43.046503 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045424 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/11f9cbee-a773-420b-9873-5c8cb7679dd8-ovn-node-metrics-cert\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.046503 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045423 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67933315-d76a-4b2c-8c92-f0249e607072-cnibin\") pod \"multus-additional-cni-plugins-bgqfs\" (UID: \"67933315-d76a-4b2c-8c92-f0249e607072\") " pod="openshift-multus/multus-additional-cni-plugins-bgqfs" Aug 04 15:56:43.046503 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045445 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-run\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:43.046503 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045488 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-run\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:43.046503 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045495 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/11f9cbee-a773-420b-9873-5c8cb7679dd8-env-overrides\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.046503 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045538 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzw72\" (UniqueName: \"kubernetes.io/projected/482db4e4-6c5f-4c1e-9666-88f470fe67db-kube-api-access-pzw72\") pod \"node-resolver-zfqf9\" (UID: \"482db4e4-6c5f-4c1e-9666-88f470fe67db\") " pod="openshift-dns/node-resolver-zfqf9" Aug 04 15:56:43.046503 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045587 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-multus-conf-dir\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.046503 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045613 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/482db4e4-6c5f-4c1e-9666-88f470fe67db-hosts-file\") pod \"node-resolver-zfqf9\" (UID: \"482db4e4-6c5f-4c1e-9666-88f470fe67db\") " pod="openshift-dns/node-resolver-zfqf9" Aug 04 15:56:43.046503 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045677 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bmm5\" (UniqueName: \"kubernetes.io/projected/cd2d6392-5d3a-49f6-be6a-954d6aaffba6-kube-api-access-6bmm5\") pod \"network-check-target-bc6jp\" (UID: \"cd2d6392-5d3a-49f6-be6a-954d6aaffba6\") " pod="openshift-network-diagnostics/network-check-target-bc6jp" Aug 04 15:56:43.046503 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045703 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/975dcd50-33ff-4dbe-b7a8-43e44e042323-konnectivity-ca\") pod \"konnectivity-agent-8jnqc\" (UID: \"975dcd50-33ff-4dbe-b7a8-43e44e042323\") " pod="kube-system/konnectivity-agent-8jnqc" Aug 04 15:56:43.047254 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045725 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c10e0ff5-2215-4032-976a-907e6dccb2d3-host\") pod \"node-ca-rz25w\" (UID: \"c10e0ff5-2215-4032-976a-907e6dccb2d3\") " pod="openshift-image-registry/node-ca-rz25w" Aug 04 15:56:43.047254 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045727 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/482db4e4-6c5f-4c1e-9666-88f470fe67db-hosts-file\") pod \"node-resolver-zfqf9\" (UID: \"482db4e4-6c5f-4c1e-9666-88f470fe67db\") " pod="openshift-dns/node-resolver-zfqf9" Aug 04 15:56:43.047254 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045748 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-etc-sysconfig\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:43.047254 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045761 2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Aug 04 15:56:43.047254 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045778 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-var-lib-kubelet\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:43.047254 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045804 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c10e0ff5-2215-4032-976a-907e6dccb2d3-host\") pod \"node-ca-rz25w\" (UID: \"c10e0ff5-2215-4032-976a-907e6dccb2d3\") " pod="openshift-image-registry/node-ca-rz25w" Aug 04 15:56:43.047254 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045816 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-node-log\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.047254 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045850 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-etc-sysconfig\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:43.047254 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045851 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-var-lib-kubelet\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:43.047254 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045858 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-node-log\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.047254 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045873 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-host-run-ovn-kubernetes\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.047254 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045897 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/11f9cbee-a773-420b-9873-5c8cb7679dd8-ovnkube-script-lib\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.047254 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045917 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e05ec517-319e-4360-8a0f-6133fb238192-original-pull-secret\") pod \"global-pull-secret-syncer-q2zlj\" (UID: \"e05ec517-319e-4360-8a0f-6133fb238192\") " pod="kube-system/global-pull-secret-syncer-q2zlj" Aug 04 15:56:43.047254 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045935 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lz94\" (UniqueName: \"kubernetes.io/projected/abba07c5-995b-4a73-8a11-cde3a8326227-kube-api-access-8lz94\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.047254 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045958 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-host-cni-netd\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.047254 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045961 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-host-run-ovn-kubernetes\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.047254 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045980 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/482db4e4-6c5f-4c1e-9666-88f470fe67db-tmp-dir\") pod \"node-resolver-zfqf9\" (UID: \"482db4e4-6c5f-4c1e-9666-88f470fe67db\") " pod="openshift-dns/node-resolver-zfqf9" Aug 04 15:56:43.047254 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.045988 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-host-cni-netd\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.048018 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046003 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-etc-kubernetes\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.048018 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046093 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/588d2847-8d2d-4f62-97f0-2253584d2e07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lccb2\" (UID: \"588d2847-8d2d-4f62-97f0-2253584d2e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lccb2" Aug 04 15:56:43.048018 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046117 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qxb4\" (UniqueName: \"kubernetes.io/projected/3932e5e3-69c2-4cee-ad2b-ac496485b140-kube-api-access-7qxb4\") pod \"iptables-alerter-kg84j\" (UID: \"3932e5e3-69c2-4cee-ad2b-ac496485b140\") " pod="openshift-network-operator/iptables-alerter-kg84j" Aug 04 15:56:43.048018 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046137 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-etc-sysctl-d\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:43.048018 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046140 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/588d2847-8d2d-4f62-97f0-2253584d2e07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lccb2\" (UID: \"588d2847-8d2d-4f62-97f0-2253584d2e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lccb2" Aug 04 15:56:43.048018 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046155 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-etc-tuned\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:43.048018 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046174 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-system-cni-dir\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.048018 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046188 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/588d2847-8d2d-4f62-97f0-2253584d2e07-sys-fs\") pod \"aws-ebs-csi-driver-node-lccb2\" (UID: \"588d2847-8d2d-4f62-97f0-2253584d2e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lccb2" Aug 04 15:56:43.048018 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046208 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-etc-openvswitch\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.048018 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046230 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-host-var-lib-cni-bin\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.048018 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046254 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-host-var-lib-cni-multus\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.048018 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046259 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/588d2847-8d2d-4f62-97f0-2253584d2e07-sys-fs\") pod \"aws-ebs-csi-driver-node-lccb2\" (UID: \"588d2847-8d2d-4f62-97f0-2253584d2e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lccb2" Aug 04 15:56:43.048018 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046242 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-etc-sysctl-d\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:43.048018 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046280 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/67933315-d76a-4b2c-8c92-f0249e607072-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bgqfs\" (UID: \"67933315-d76a-4b2c-8c92-f0249e607072\") " pod="openshift-multus/multus-additional-cni-plugins-bgqfs" Aug 04 15:56:43.048018 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046287 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/482db4e4-6c5f-4c1e-9666-88f470fe67db-tmp-dir\") pod \"node-resolver-zfqf9\" (UID: \"482db4e4-6c5f-4c1e-9666-88f470fe67db\") " pod="openshift-dns/node-resolver-zfqf9" Aug 04 15:56:43.048018 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046308 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/67933315-d76a-4b2c-8c92-f0249e607072-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bgqfs\" (UID: \"67933315-d76a-4b2c-8c92-f0249e607072\") " pod="openshift-multus/multus-additional-cni-plugins-bgqfs" Aug 04 15:56:43.048018 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046315 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-etc-openvswitch\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.048747 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046338 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3932e5e3-69c2-4cee-ad2b-ac496485b140-iptables-alerter-script\") pod \"iptables-alerter-kg84j\" (UID: \"3932e5e3-69c2-4cee-ad2b-ac496485b140\") " pod="openshift-network-operator/iptables-alerter-kg84j" Aug 04 15:56:43.048747 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046382 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3932e5e3-69c2-4cee-ad2b-ac496485b140-host-slash\") pod \"iptables-alerter-kg84j\" (UID: \"3932e5e3-69c2-4cee-ad2b-ac496485b140\") " pod="openshift-network-operator/iptables-alerter-kg84j" Aug 04 15:56:43.048747 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046435 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-host-kubelet\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.048747 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046457 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-systemd-units\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.048747 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046479 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/11f9cbee-a773-420b-9873-5c8cb7679dd8-ovnkube-config\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.048747 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046485 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/11f9cbee-a773-420b-9873-5c8cb7679dd8-ovnkube-script-lib\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.048747 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046507 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-etc-modprobe-d\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:43.048747 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046564 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-host-kubelet\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.048747 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046589 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-log-socket\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.048747 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046609 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-host-cni-bin\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.048747 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046629 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-sys\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:43.048747 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046650 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-tmp\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:43.048747 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046791 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/67933315-d76a-4b2c-8c92-f0249e607072-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bgqfs\" (UID: \"67933315-d76a-4b2c-8c92-f0249e607072\") " pod="openshift-multus/multus-additional-cni-plugins-bgqfs" Aug 04 15:56:43.048747 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046835 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/67933315-d76a-4b2c-8c92-f0249e607072-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bgqfs\" (UID: \"67933315-d76a-4b2c-8c92-f0249e607072\") " pod="openshift-multus/multus-additional-cni-plugins-bgqfs" Aug 04 15:56:43.048747 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046843 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3932e5e3-69c2-4cee-ad2b-ac496485b140-iptables-alerter-script\") pod \"iptables-alerter-kg84j\" (UID: \"3932e5e3-69c2-4cee-ad2b-ac496485b140\") " pod="openshift-network-operator/iptables-alerter-kg84j" Aug 04 15:56:43.048747 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046846 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-log-socket\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.048747 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046564 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-systemd-units\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.049364 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046887 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3932e5e3-69c2-4cee-ad2b-ac496485b140-host-slash\") pod \"iptables-alerter-kg84j\" (UID: \"3932e5e3-69c2-4cee-ad2b-ac496485b140\") " pod="openshift-network-operator/iptables-alerter-kg84j" Aug 04 15:56:43.049364 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046889 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11f9cbee-a773-420b-9873-5c8cb7679dd8-host-cni-bin\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.049364 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046948 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-sys\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:43.049364 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.046956 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-etc-modprobe-d\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:43.049364 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.047018 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/11f9cbee-a773-420b-9873-5c8cb7679dd8-ovnkube-config\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.049364 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.049336 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-etc-tuned\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:43.049668 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.049475 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/11f9cbee-a773-420b-9873-5c8cb7679dd8-ovn-node-metrics-cert\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.049721 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.049709 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-tmp\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:43.052924 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:43.052900 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 15:56:43.052924 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:43.052927 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 15:56:43.053084 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:43.052942 2573 projected.go:196] Error preparing data for projected volume kube-api-access-6bmm5 for pod openshift-network-diagnostics/network-check-target-bc6jp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:56:43.053084 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.053007 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8b28\" (UniqueName: \"kubernetes.io/projected/67933315-d76a-4b2c-8c92-f0249e607072-kube-api-access-q8b28\") pod \"multus-additional-cni-plugins-bgqfs\" (UID: \"67933315-d76a-4b2c-8c92-f0249e607072\") " pod="openshift-multus/multus-additional-cni-plugins-bgqfs" Aug 04 15:56:43.053084 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:43.053021 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd2d6392-5d3a-49f6-be6a-954d6aaffba6-kube-api-access-6bmm5 podName:cd2d6392-5d3a-49f6-be6a-954d6aaffba6 nodeName:}" failed. No retries permitted until 2026-08-04 15:56:43.55298822 +0000 UTC m=+2.970530327 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6bmm5" (UniqueName: "kubernetes.io/projected/cd2d6392-5d3a-49f6-be6a-954d6aaffba6-kube-api-access-6bmm5") pod "network-check-target-bc6jp" (UID: "cd2d6392-5d3a-49f6-be6a-954d6aaffba6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:56:43.053235 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.053155 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj5mb\" (UniqueName: \"kubernetes.io/projected/ef05b1ee-7f9a-42a0-825d-55ba18709e5f-kube-api-access-gj5mb\") pod \"tuned-zcx94\" (UID: \"ef05b1ee-7f9a-42a0-825d-55ba18709e5f\") " pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:43.054420 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.054396 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzw72\" (UniqueName: \"kubernetes.io/projected/482db4e4-6c5f-4c1e-9666-88f470fe67db-kube-api-access-pzw72\") pod \"node-resolver-zfqf9\" (UID: \"482db4e4-6c5f-4c1e-9666-88f470fe67db\") " pod="openshift-dns/node-resolver-zfqf9" Aug 04 15:56:43.054732 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.054602 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pq7k\" (UniqueName: \"kubernetes.io/projected/588d2847-8d2d-4f62-97f0-2253584d2e07-kube-api-access-2pq7k\") pod \"aws-ebs-csi-driver-node-lccb2\" (UID: \"588d2847-8d2d-4f62-97f0-2253584d2e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lccb2" Aug 04 15:56:43.055291 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.055270 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkjf5\" (UniqueName: \"kubernetes.io/projected/c10e0ff5-2215-4032-976a-907e6dccb2d3-kube-api-access-pkjf5\") pod \"node-ca-rz25w\" (UID: \"c10e0ff5-2215-4032-976a-907e6dccb2d3\") " pod="openshift-image-registry/node-ca-rz25w" Aug 04 15:56:43.055842 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.055822 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qxb4\" (UniqueName: \"kubernetes.io/projected/3932e5e3-69c2-4cee-ad2b-ac496485b140-kube-api-access-7qxb4\") pod \"iptables-alerter-kg84j\" (UID: \"3932e5e3-69c2-4cee-ad2b-ac496485b140\") " pod="openshift-network-operator/iptables-alerter-kg84j" Aug 04 15:56:43.056177 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.056157 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28b7m\" (UniqueName: \"kubernetes.io/projected/11f9cbee-a773-420b-9873-5c8cb7679dd8-kube-api-access-28b7m\") pod \"ovnkube-node-mtbc5\" (UID: \"11f9cbee-a773-420b-9873-5c8cb7679dd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.147161 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147130 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-multus-socket-dir-parent\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.147314 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147165 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-host-run-multus-certs\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.147314 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147191 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-os-release\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.147314 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147216 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e05ec517-319e-4360-8a0f-6133fb238192-kubelet-config\") pod \"global-pull-secret-syncer-q2zlj\" (UID: \"e05ec517-319e-4360-8a0f-6133fb238192\") " pod="kube-system/global-pull-secret-syncer-q2zlj" Aug 04 15:56:43.147314 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147238 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb454294-c3a4-4d8d-b705-95350ed95c13-metrics-certs\") pod \"network-metrics-daemon-rcb4z\" (UID: \"fb454294-c3a4-4d8d-b705-95350ed95c13\") " pod="openshift-multus/network-metrics-daemon-rcb4z" Aug 04 15:56:43.147314 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147245 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-host-run-multus-certs\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.147314 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147259 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-host-run-k8s-cni-cncf-io\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.147637 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:43.147365 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:56:43.147637 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147391 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-multus-conf-dir\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.147637 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147427 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/975dcd50-33ff-4dbe-b7a8-43e44e042323-konnectivity-ca\") pod \"konnectivity-agent-8jnqc\" (UID: \"975dcd50-33ff-4dbe-b7a8-43e44e042323\") " pod="kube-system/konnectivity-agent-8jnqc" Aug 04 15:56:43.147637 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147343 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-os-release\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.147637 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:43.147439 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb454294-c3a4-4d8d-b705-95350ed95c13-metrics-certs podName:fb454294-c3a4-4d8d-b705-95350ed95c13 nodeName:}" failed. No retries permitted until 2026-08-04 15:56:43.647419347 +0000 UTC m=+3.064961468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb454294-c3a4-4d8d-b705-95350ed95c13-metrics-certs") pod "network-metrics-daemon-rcb4z" (UID: "fb454294-c3a4-4d8d-b705-95350ed95c13") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:56:43.147637 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147337 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e05ec517-319e-4360-8a0f-6133fb238192-kubelet-config\") pod \"global-pull-secret-syncer-q2zlj\" (UID: \"e05ec517-319e-4360-8a0f-6133fb238192\") " pod="kube-system/global-pull-secret-syncer-q2zlj" Aug 04 15:56:43.147637 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147248 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-multus-socket-dir-parent\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.147637 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147305 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-host-run-k8s-cni-cncf-io\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.147637 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147484 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e05ec517-319e-4360-8a0f-6133fb238192-original-pull-secret\") pod \"global-pull-secret-syncer-q2zlj\" (UID: \"e05ec517-319e-4360-8a0f-6133fb238192\") " pod="kube-system/global-pull-secret-syncer-q2zlj" Aug 04 15:56:43.147637 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147504 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lz94\" (UniqueName: \"kubernetes.io/projected/abba07c5-995b-4a73-8a11-cde3a8326227-kube-api-access-8lz94\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.147637 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147519 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-etc-kubernetes\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.147637 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147544 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-system-cni-dir\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.147637 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147469 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-multus-conf-dir\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.147637 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147576 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-etc-kubernetes\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.147637 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147582 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-host-var-lib-cni-bin\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.147637 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:43.147574 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 15:56:43.147637 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147615 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-system-cni-dir\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.147637 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147617 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-host-var-lib-cni-bin\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.148370 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147632 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-host-var-lib-cni-multus\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.148370 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:43.147661 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e05ec517-319e-4360-8a0f-6133fb238192-original-pull-secret podName:e05ec517-319e-4360-8a0f-6133fb238192 nodeName:}" failed. No retries permitted until 2026-08-04 15:56:43.647647464 +0000 UTC m=+3.065189587 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e05ec517-319e-4360-8a0f-6133fb238192-original-pull-secret") pod "global-pull-secret-syncer-q2zlj" (UID: "e05ec517-319e-4360-8a0f-6133fb238192") : object "kube-system"/"original-pull-secret" not registered Aug 04 15:56:43.148370 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147672 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-host-var-lib-cni-multus\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.148370 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147690 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp5cx\" (UniqueName: \"kubernetes.io/projected/fb454294-c3a4-4d8d-b705-95350ed95c13-kube-api-access-lp5cx\") pod \"network-metrics-daemon-rcb4z\" (UID: \"fb454294-c3a4-4d8d-b705-95350ed95c13\") " pod="openshift-multus/network-metrics-daemon-rcb4z" Aug 04 15:56:43.148370 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147710 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-host-var-lib-kubelet\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.148370 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147726 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/abba07c5-995b-4a73-8a11-cde3a8326227-multus-daemon-config\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.148370 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147744 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-hostroot\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.148370 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147760 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-host-run-netns\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.148370 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147776 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/975dcd50-33ff-4dbe-b7a8-43e44e042323-agent-certs\") pod \"konnectivity-agent-8jnqc\" (UID: \"975dcd50-33ff-4dbe-b7a8-43e44e042323\") " pod="kube-system/konnectivity-agent-8jnqc" Aug 04 15:56:43.148370 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147797 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e05ec517-319e-4360-8a0f-6133fb238192-dbus\") pod \"global-pull-secret-syncer-q2zlj\" (UID: \"e05ec517-319e-4360-8a0f-6133fb238192\") " pod="kube-system/global-pull-secret-syncer-q2zlj" Aug 04 15:56:43.148370 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147803 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-host-run-netns\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.148370 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147811 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-multus-cni-dir\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.148370 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147807 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-hostroot\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.148370 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147840 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-cnibin\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.148370 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147852 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-host-var-lib-kubelet\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.148370 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147897 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-multus-cni-dir\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.148370 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147937 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/abba07c5-995b-4a73-8a11-cde3a8326227-cni-binary-copy\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.149164 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147960 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/975dcd50-33ff-4dbe-b7a8-43e44e042323-konnectivity-ca\") pod \"konnectivity-agent-8jnqc\" (UID: \"975dcd50-33ff-4dbe-b7a8-43e44e042323\") " pod="kube-system/konnectivity-agent-8jnqc" Aug 04 15:56:43.149164 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.147967 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e05ec517-319e-4360-8a0f-6133fb238192-dbus\") pod \"global-pull-secret-syncer-q2zlj\" (UID: \"e05ec517-319e-4360-8a0f-6133fb238192\") " pod="kube-system/global-pull-secret-syncer-q2zlj" Aug 04 15:56:43.149164 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.148144 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/abba07c5-995b-4a73-8a11-cde3a8326227-cnibin\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.149164 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.148411 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/abba07c5-995b-4a73-8a11-cde3a8326227-cni-binary-copy\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.149164 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.148477 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/abba07c5-995b-4a73-8a11-cde3a8326227-multus-daemon-config\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.150222 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.150204 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/975dcd50-33ff-4dbe-b7a8-43e44e042323-agent-certs\") pod \"konnectivity-agent-8jnqc\" (UID: \"975dcd50-33ff-4dbe-b7a8-43e44e042323\") " pod="kube-system/konnectivity-agent-8jnqc" Aug 04 15:56:43.157776 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.157727 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lz94\" (UniqueName: \"kubernetes.io/projected/abba07c5-995b-4a73-8a11-cde3a8326227-kube-api-access-8lz94\") pod \"multus-jbkjx\" (UID: \"abba07c5-995b-4a73-8a11-cde3a8326227\") " pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.157870 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.157822 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp5cx\" (UniqueName: \"kubernetes.io/projected/fb454294-c3a4-4d8d-b705-95350ed95c13-kube-api-access-lp5cx\") pod \"network-metrics-daemon-rcb4z\" (UID: \"fb454294-c3a4-4d8d-b705-95350ed95c13\") " pod="openshift-multus/network-metrics-daemon-rcb4z" Aug 04 15:56:43.236803 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.236773 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lccb2" Aug 04 15:56:43.244275 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.244255 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bgqfs" Aug 04 15:56:43.244722 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:43.244696 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod588d2847_8d2d_4f62_97f0_2253584d2e07.slice/crio-ddd326fe418cecd884284bad2029316500d9d0a2002030381dc7fed22b940e88 WatchSource:0}: Error finding container ddd326fe418cecd884284bad2029316500d9d0a2002030381dc7fed22b940e88: Status 404 returned error can't find the container with id ddd326fe418cecd884284bad2029316500d9d0a2002030381dc7fed22b940e88 Aug 04 15:56:43.251415 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:43.251393 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67933315_d76a_4b2c_8c92_f0249e607072.slice/crio-86728318a912b142b9bfac160e01388ab50769b0a4dde9b2fa1500ee97c695e1 WatchSource:0}: Error finding container 86728318a912b142b9bfac160e01388ab50769b0a4dde9b2fa1500ee97c695e1: Status 404 returned error can't find the container with id 86728318a912b142b9bfac160e01388ab50769b0a4dde9b2fa1500ee97c695e1 Aug 04 15:56:43.252013 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.251990 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kg84j" Aug 04 15:56:43.256706 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.256686 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zfqf9" Aug 04 15:56:43.260017 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:43.259991 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3932e5e3_69c2_4cee_ad2b_ac496485b140.slice/crio-6eae19016cd1088f6ac4d58e8f14947aea7d74863226cc3c85d7f0eb05852ee6 WatchSource:0}: Error finding container 6eae19016cd1088f6ac4d58e8f14947aea7d74863226cc3c85d7f0eb05852ee6: Status 404 returned error can't find the container with id 6eae19016cd1088f6ac4d58e8f14947aea7d74863226cc3c85d7f0eb05852ee6 Aug 04 15:56:43.263465 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.263445 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:56:43.266507 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:43.266476 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod482db4e4_6c5f_4c1e_9666_88f470fe67db.slice/crio-11014223363c26a7f4c5eb7517503a0f496137cabb0c79bd887cda8fac8e4587 WatchSource:0}: Error finding container 11014223363c26a7f4c5eb7517503a0f496137cabb0c79bd887cda8fac8e4587: Status 404 returned error can't find the container with id 11014223363c26a7f4c5eb7517503a0f496137cabb0c79bd887cda8fac8e4587 Aug 04 15:56:43.268383 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.268365 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rz25w" Aug 04 15:56:43.271859 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:43.271834 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f9cbee_a773_420b_9873_5c8cb7679dd8.slice/crio-7cfb6698aa5bb5941f7cf5ab1347398203e9e2295188874c2f4ed7f15c2271b6 WatchSource:0}: Error finding container 7cfb6698aa5bb5941f7cf5ab1347398203e9e2295188874c2f4ed7f15c2271b6: Status 404 returned error can't find the container with id 7cfb6698aa5bb5941f7cf5ab1347398203e9e2295188874c2f4ed7f15c2271b6 Aug 04 15:56:43.274714 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.274695 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zcx94" Aug 04 15:56:43.278106 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:43.278086 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc10e0ff5_2215_4032_976a_907e6dccb2d3.slice/crio-348ae59b33c2a742be1bb444613ab0b60c68c3f721dbf2325fc3994b78bc0274 WatchSource:0}: Error finding container 348ae59b33c2a742be1bb444613ab0b60c68c3f721dbf2325fc3994b78bc0274: Status 404 returned error can't find the container with id 348ae59b33c2a742be1bb444613ab0b60c68c3f721dbf2325fc3994b78bc0274 Aug 04 15:56:43.279824 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.279797 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8jnqc" Aug 04 15:56:43.282601 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:43.282575 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef05b1ee_7f9a_42a0_825d_55ba18709e5f.slice/crio-08075bba16e621ec3166a5c59751aa1af0ebfe767daca9123030c351476e1839 WatchSource:0}: Error finding container 08075bba16e621ec3166a5c59751aa1af0ebfe767daca9123030c351476e1839: Status 404 returned error can't find the container with id 08075bba16e621ec3166a5c59751aa1af0ebfe767daca9123030c351476e1839 Aug 04 15:56:43.285387 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.285368 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jbkjx" Aug 04 15:56:43.287602 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:43.287576 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod975dcd50_33ff_4dbe_b7a8_43e44e042323.slice/crio-b9555b942509c1ca1939ae8964d843616dd8ead7e22a3bdb1bfb71f914e56b83 WatchSource:0}: Error finding container b9555b942509c1ca1939ae8964d843616dd8ead7e22a3bdb1bfb71f914e56b83: Status 404 returned error can't find the container with id b9555b942509c1ca1939ae8964d843616dd8ead7e22a3bdb1bfb71f914e56b83 Aug 04 15:56:43.292755 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:56:43.292731 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabba07c5_995b_4a73_8a11_cde3a8326227.slice/crio-8a2520adf4ae6c6adca1bd176f4aaad33aa497ac9e1737705f14e920c0bca87f WatchSource:0}: Error finding container 8a2520adf4ae6c6adca1bd176f4aaad33aa497ac9e1737705f14e920c0bca87f: Status 404 returned error can't find the container with id 8a2520adf4ae6c6adca1bd176f4aaad33aa497ac9e1737705f14e920c0bca87f Aug 04 15:56:43.652306 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.652239 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb454294-c3a4-4d8d-b705-95350ed95c13-metrics-certs\") pod \"network-metrics-daemon-rcb4z\" (UID: \"fb454294-c3a4-4d8d-b705-95350ed95c13\") " pod="openshift-multus/network-metrics-daemon-rcb4z" Aug 04 15:56:43.652452 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.652343 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bmm5\" (UniqueName: \"kubernetes.io/projected/cd2d6392-5d3a-49f6-be6a-954d6aaffba6-kube-api-access-6bmm5\") pod \"network-check-target-bc6jp\" (UID: \"cd2d6392-5d3a-49f6-be6a-954d6aaffba6\") " pod="openshift-network-diagnostics/network-check-target-bc6jp" Aug 04 15:56:43.652452 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:43.652394 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:56:43.652584 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:43.652458 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb454294-c3a4-4d8d-b705-95350ed95c13-metrics-certs podName:fb454294-c3a4-4d8d-b705-95350ed95c13 nodeName:}" failed. No retries permitted until 2026-08-04 15:56:44.652443482 +0000 UTC m=+4.069985587 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb454294-c3a4-4d8d-b705-95350ed95c13-metrics-certs") pod "network-metrics-daemon-rcb4z" (UID: "fb454294-c3a4-4d8d-b705-95350ed95c13") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:56:43.652584 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:43.652487 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 15:56:43.652584 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:43.652491 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 15:56:43.652584 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:43.652512 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 15:56:43.652584 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:43.652524 2573 projected.go:196] Error preparing data for projected volume kube-api-access-6bmm5 for pod openshift-network-diagnostics/network-check-target-bc6jp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:56:43.652584 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:43.652542 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e05ec517-319e-4360-8a0f-6133fb238192-original-pull-secret podName:e05ec517-319e-4360-8a0f-6133fb238192 nodeName:}" failed. No retries permitted until 2026-08-04 15:56:44.652528339 +0000 UTC m=+4.070070444 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e05ec517-319e-4360-8a0f-6133fb238192-original-pull-secret") pod "global-pull-secret-syncer-q2zlj" (UID: "e05ec517-319e-4360-8a0f-6133fb238192") : object "kube-system"/"original-pull-secret" not registered Aug 04 15:56:43.652584 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.652381 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e05ec517-319e-4360-8a0f-6133fb238192-original-pull-secret\") pod \"global-pull-secret-syncer-q2zlj\" (UID: \"e05ec517-319e-4360-8a0f-6133fb238192\") " pod="kube-system/global-pull-secret-syncer-q2zlj" Aug 04 15:56:43.652584 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:43.652588 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd2d6392-5d3a-49f6-be6a-954d6aaffba6-kube-api-access-6bmm5 podName:cd2d6392-5d3a-49f6-be6a-954d6aaffba6 nodeName:}" failed. No retries permitted until 2026-08-04 15:56:44.65257493 +0000 UTC m=+4.070117037 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6bmm5" (UniqueName: "kubernetes.io/projected/cd2d6392-5d3a-49f6-be6a-954d6aaffba6-kube-api-access-6bmm5") pod "network-check-target-bc6jp" (UID: "cd2d6392-5d3a-49f6-be6a-954d6aaffba6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:56:43.989413 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.989171 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 15:51:42 +0000 UTC" deadline="2028-01-23 01:13:36.819570823 +0000 UTC" Aug 04 15:56:43.989413 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:43.989415 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12873h16m52.830161972s" Aug 04 15:56:44.085603 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:44.085504 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rz25w" event={"ID":"c10e0ff5-2215-4032-976a-907e6dccb2d3","Type":"ContainerStarted","Data":"348ae59b33c2a742be1bb444613ab0b60c68c3f721dbf2325fc3994b78bc0274"} Aug 04 15:56:44.088327 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:44.088298 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" event={"ID":"11f9cbee-a773-420b-9873-5c8cb7679dd8","Type":"ContainerStarted","Data":"7cfb6698aa5bb5941f7cf5ab1347398203e9e2295188874c2f4ed7f15c2271b6"} Aug 04 15:56:44.097733 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:44.097619 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zfqf9" event={"ID":"482db4e4-6c5f-4c1e-9666-88f470fe67db","Type":"ContainerStarted","Data":"11014223363c26a7f4c5eb7517503a0f496137cabb0c79bd887cda8fac8e4587"} Aug 04 15:56:44.107559 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:44.107512 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kg84j" event={"ID":"3932e5e3-69c2-4cee-ad2b-ac496485b140","Type":"ContainerStarted","Data":"6eae19016cd1088f6ac4d58e8f14947aea7d74863226cc3c85d7f0eb05852ee6"} Aug 04 15:56:44.109899 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:44.109857 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jbkjx" event={"ID":"abba07c5-995b-4a73-8a11-cde3a8326227","Type":"ContainerStarted","Data":"8a2520adf4ae6c6adca1bd176f4aaad33aa497ac9e1737705f14e920c0bca87f"} Aug 04 15:56:44.114518 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:44.114488 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgqfs" event={"ID":"67933315-d76a-4b2c-8c92-f0249e607072","Type":"ContainerStarted","Data":"86728318a912b142b9bfac160e01388ab50769b0a4dde9b2fa1500ee97c695e1"} Aug 04 15:56:44.117297 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:44.117245 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lccb2" event={"ID":"588d2847-8d2d-4f62-97f0-2253584d2e07","Type":"ContainerStarted","Data":"ddd326fe418cecd884284bad2029316500d9d0a2002030381dc7fed22b940e88"} Aug 04 15:56:44.120739 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:44.120710 2573 generic.go:356] "Generic (PLEG): container finished" podID="5e35ae5863cdf97bdb5252257e0c413f" containerID="1d9da8c50e7e75474685ed8bcaccd772338e6fc5cdc04b44143cd75666f3e22b" exitCode=0 Aug 04 15:56:44.121306 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:44.121281 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-210.ec2.internal" event={"ID":"5e35ae5863cdf97bdb5252257e0c413f","Type":"ContainerDied","Data":"1d9da8c50e7e75474685ed8bcaccd772338e6fc5cdc04b44143cd75666f3e22b"} Aug 04 15:56:44.133022 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:44.132995 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-210.ec2.internal" event={"ID":"fd26de1bd1ac6d99210ec0097a02c100","Type":"ContainerStarted","Data":"db3fd23bb4e4ec48a32faff609347c210383d43bf77589a09e2abd53c643a24b"} Aug 04 15:56:44.142188 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:44.142149 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8jnqc" event={"ID":"975dcd50-33ff-4dbe-b7a8-43e44e042323","Type":"ContainerStarted","Data":"b9555b942509c1ca1939ae8964d843616dd8ead7e22a3bdb1bfb71f914e56b83"} Aug 04 15:56:44.149034 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:44.148989 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zcx94" event={"ID":"ef05b1ee-7f9a-42a0-825d-55ba18709e5f","Type":"ContainerStarted","Data":"08075bba16e621ec3166a5c59751aa1af0ebfe767daca9123030c351476e1839"} Aug 04 15:56:44.662491 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:44.662390 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb454294-c3a4-4d8d-b705-95350ed95c13-metrics-certs\") pod \"network-metrics-daemon-rcb4z\" (UID: \"fb454294-c3a4-4d8d-b705-95350ed95c13\") " pod="openshift-multus/network-metrics-daemon-rcb4z" Aug 04 15:56:44.662491 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:44.662445 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bmm5\" (UniqueName: \"kubernetes.io/projected/cd2d6392-5d3a-49f6-be6a-954d6aaffba6-kube-api-access-6bmm5\") pod \"network-check-target-bc6jp\" (UID: \"cd2d6392-5d3a-49f6-be6a-954d6aaffba6\") " pod="openshift-network-diagnostics/network-check-target-bc6jp" Aug 04 15:56:44.662491 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:44.662479 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e05ec517-319e-4360-8a0f-6133fb238192-original-pull-secret\") pod \"global-pull-secret-syncer-q2zlj\" (UID: \"e05ec517-319e-4360-8a0f-6133fb238192\") " pod="kube-system/global-pull-secret-syncer-q2zlj" Aug 04 15:56:44.662795 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:44.662637 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 15:56:44.662795 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:44.662705 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e05ec517-319e-4360-8a0f-6133fb238192-original-pull-secret podName:e05ec517-319e-4360-8a0f-6133fb238192 nodeName:}" failed. No retries permitted until 2026-08-04 15:56:46.662684469 +0000 UTC m=+6.080226578 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e05ec517-319e-4360-8a0f-6133fb238192-original-pull-secret") pod "global-pull-secret-syncer-q2zlj" (UID: "e05ec517-319e-4360-8a0f-6133fb238192") : object "kube-system"/"original-pull-secret" not registered Aug 04 15:56:44.663268 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:44.663139 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:56:44.663268 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:44.663194 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb454294-c3a4-4d8d-b705-95350ed95c13-metrics-certs podName:fb454294-c3a4-4d8d-b705-95350ed95c13 nodeName:}" failed. No retries permitted until 2026-08-04 15:56:46.663177975 +0000 UTC m=+6.080720084 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb454294-c3a4-4d8d-b705-95350ed95c13-metrics-certs") pod "network-metrics-daemon-rcb4z" (UID: "fb454294-c3a4-4d8d-b705-95350ed95c13") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:56:44.663268 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:44.663275 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 15:56:44.663484 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:44.663293 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 15:56:44.663484 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:44.663307 2573 projected.go:196] Error preparing data for projected volume kube-api-access-6bmm5 for pod openshift-network-diagnostics/network-check-target-bc6jp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:56:44.663484 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:44.663345 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd2d6392-5d3a-49f6-be6a-954d6aaffba6-kube-api-access-6bmm5 podName:cd2d6392-5d3a-49f6-be6a-954d6aaffba6 nodeName:}" failed. No retries permitted until 2026-08-04 15:56:46.663333452 +0000 UTC m=+6.080875559 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6bmm5" (UniqueName: "kubernetes.io/projected/cd2d6392-5d3a-49f6-be6a-954d6aaffba6-kube-api-access-6bmm5") pod "network-check-target-bc6jp" (UID: "cd2d6392-5d3a-49f6-be6a-954d6aaffba6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:56:44.990278 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:44.990164 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 15:51:42 +0000 UTC" deadline="2028-01-12 00:40:46.691935078 +0000 UTC" Aug 04 15:56:44.990278 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:44.990227 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12608h44m1.701713009s" Aug 04 15:56:45.073071 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:45.072567 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc6jp" Aug 04 15:56:45.073071 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:45.072569 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zlj" Aug 04 15:56:45.073071 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:45.072694 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bc6jp" podUID="cd2d6392-5d3a-49f6-be6a-954d6aaffba6" Aug 04 15:56:45.073071 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:45.072718 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcb4z" Aug 04 15:56:45.073071 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:45.072817 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2zlj" podUID="e05ec517-319e-4360-8a0f-6133fb238192" Aug 04 15:56:45.073071 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:45.072904 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcb4z" podUID="fb454294-c3a4-4d8d-b705-95350ed95c13" Aug 04 15:56:45.168571 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:45.168519 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-210.ec2.internal" event={"ID":"5e35ae5863cdf97bdb5252257e0c413f","Type":"ContainerStarted","Data":"a26334ecaa8c88e379c2e61f2454b3931783efcdd3f13d838c7f31042b8870fb"} Aug 04 15:56:46.679447 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:46.678758 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb454294-c3a4-4d8d-b705-95350ed95c13-metrics-certs\") pod \"network-metrics-daemon-rcb4z\" (UID: \"fb454294-c3a4-4d8d-b705-95350ed95c13\") " pod="openshift-multus/network-metrics-daemon-rcb4z" Aug 04 15:56:46.679447 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:46.678829 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bmm5\" (UniqueName: \"kubernetes.io/projected/cd2d6392-5d3a-49f6-be6a-954d6aaffba6-kube-api-access-6bmm5\") pod \"network-check-target-bc6jp\" (UID: \"cd2d6392-5d3a-49f6-be6a-954d6aaffba6\") " pod="openshift-network-diagnostics/network-check-target-bc6jp" Aug 04 15:56:46.679447 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:46.678869 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e05ec517-319e-4360-8a0f-6133fb238192-original-pull-secret\") pod \"global-pull-secret-syncer-q2zlj\" (UID: \"e05ec517-319e-4360-8a0f-6133fb238192\") " pod="kube-system/global-pull-secret-syncer-q2zlj" Aug 04 15:56:46.679447 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:46.679016 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 15:56:46.679447 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:46.679040 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:56:46.679447 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:46.679080 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e05ec517-319e-4360-8a0f-6133fb238192-original-pull-secret podName:e05ec517-319e-4360-8a0f-6133fb238192 nodeName:}" failed. No retries permitted until 2026-08-04 15:56:50.679062143 +0000 UTC m=+10.096604266 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e05ec517-319e-4360-8a0f-6133fb238192-original-pull-secret") pod "global-pull-secret-syncer-q2zlj" (UID: "e05ec517-319e-4360-8a0f-6133fb238192") : object "kube-system"/"original-pull-secret" not registered Aug 04 15:56:46.679447 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:46.679099 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb454294-c3a4-4d8d-b705-95350ed95c13-metrics-certs podName:fb454294-c3a4-4d8d-b705-95350ed95c13 nodeName:}" failed. No retries permitted until 2026-08-04 15:56:50.679090597 +0000 UTC m=+10.096632701 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb454294-c3a4-4d8d-b705-95350ed95c13-metrics-certs") pod "network-metrics-daemon-rcb4z" (UID: "fb454294-c3a4-4d8d-b705-95350ed95c13") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:56:46.679447 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:46.679113 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 15:56:46.679447 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:46.679146 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 15:56:46.679447 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:46.679160 2573 projected.go:196] Error preparing data for projected volume kube-api-access-6bmm5 for pod openshift-network-diagnostics/network-check-target-bc6jp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:56:46.679447 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:46.679213 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd2d6392-5d3a-49f6-be6a-954d6aaffba6-kube-api-access-6bmm5 podName:cd2d6392-5d3a-49f6-be6a-954d6aaffba6 nodeName:}" failed. No retries permitted until 2026-08-04 15:56:50.679196058 +0000 UTC m=+10.096738177 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6bmm5" (UniqueName: "kubernetes.io/projected/cd2d6392-5d3a-49f6-be6a-954d6aaffba6-kube-api-access-6bmm5") pod "network-check-target-bc6jp" (UID: "cd2d6392-5d3a-49f6-be6a-954d6aaffba6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:56:47.073683 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:47.073291 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zlj" Aug 04 15:56:47.073683 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:47.073335 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcb4z" Aug 04 15:56:47.073683 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:47.073290 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc6jp" Aug 04 15:56:47.073683 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:47.073432 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2zlj" podUID="e05ec517-319e-4360-8a0f-6133fb238192" Aug 04 15:56:47.073683 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:47.073533 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bc6jp" podUID="cd2d6392-5d3a-49f6-be6a-954d6aaffba6" Aug 04 15:56:47.073683 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:47.073631 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcb4z" podUID="fb454294-c3a4-4d8d-b705-95350ed95c13" Aug 04 15:56:49.072611 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:49.072559 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc6jp" Aug 04 15:56:49.073077 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:49.072560 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zlj" Aug 04 15:56:49.073077 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:49.072683 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcb4z" Aug 04 15:56:49.073077 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:49.072704 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bc6jp" podUID="cd2d6392-5d3a-49f6-be6a-954d6aaffba6" Aug 04 15:56:49.073077 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:49.072819 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcb4z" podUID="fb454294-c3a4-4d8d-b705-95350ed95c13" Aug 04 15:56:49.073077 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:49.072901 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2zlj" podUID="e05ec517-319e-4360-8a0f-6133fb238192" Aug 04 15:56:50.710629 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:50.710589 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb454294-c3a4-4d8d-b705-95350ed95c13-metrics-certs\") pod \"network-metrics-daemon-rcb4z\" (UID: \"fb454294-c3a4-4d8d-b705-95350ed95c13\") " pod="openshift-multus/network-metrics-daemon-rcb4z" Aug 04 15:56:50.711110 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:50.710649 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bmm5\" (UniqueName: \"kubernetes.io/projected/cd2d6392-5d3a-49f6-be6a-954d6aaffba6-kube-api-access-6bmm5\") pod \"network-check-target-bc6jp\" (UID: \"cd2d6392-5d3a-49f6-be6a-954d6aaffba6\") " pod="openshift-network-diagnostics/network-check-target-bc6jp" Aug 04 15:56:50.711110 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:50.710684 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e05ec517-319e-4360-8a0f-6133fb238192-original-pull-secret\") pod \"global-pull-secret-syncer-q2zlj\" (UID: \"e05ec517-319e-4360-8a0f-6133fb238192\") " pod="kube-system/global-pull-secret-syncer-q2zlj" Aug 04 15:56:50.711110 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:50.710788 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:56:50.711110 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:50.710837 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 15:56:50.711110 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:50.710863 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb454294-c3a4-4d8d-b705-95350ed95c13-metrics-certs podName:fb454294-c3a4-4d8d-b705-95350ed95c13 nodeName:}" failed. No retries permitted until 2026-08-04 15:56:58.710847217 +0000 UTC m=+18.128389325 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb454294-c3a4-4d8d-b705-95350ed95c13-metrics-certs") pod "network-metrics-daemon-rcb4z" (UID: "fb454294-c3a4-4d8d-b705-95350ed95c13") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:56:50.711110 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:50.710889 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e05ec517-319e-4360-8a0f-6133fb238192-original-pull-secret podName:e05ec517-319e-4360-8a0f-6133fb238192 nodeName:}" failed. No retries permitted until 2026-08-04 15:56:58.710873274 +0000 UTC m=+18.128415380 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e05ec517-319e-4360-8a0f-6133fb238192-original-pull-secret") pod "global-pull-secret-syncer-q2zlj" (UID: "e05ec517-319e-4360-8a0f-6133fb238192") : object "kube-system"/"original-pull-secret" not registered Aug 04 15:56:50.711110 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:50.710966 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 15:56:50.711110 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:50.710982 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 15:56:50.711110 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:50.710995 2573 projected.go:196] Error preparing data for projected volume kube-api-access-6bmm5 for pod openshift-network-diagnostics/network-check-target-bc6jp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:56:50.711110 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:50.711039 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd2d6392-5d3a-49f6-be6a-954d6aaffba6-kube-api-access-6bmm5 podName:cd2d6392-5d3a-49f6-be6a-954d6aaffba6 nodeName:}" failed. No retries permitted until 2026-08-04 15:56:58.711028161 +0000 UTC m=+18.128570265 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6bmm5" (UniqueName: "kubernetes.io/projected/cd2d6392-5d3a-49f6-be6a-954d6aaffba6-kube-api-access-6bmm5") pod "network-check-target-bc6jp" (UID: "cd2d6392-5d3a-49f6-be6a-954d6aaffba6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:56:51.073493 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:51.073414 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc6jp" Aug 04 15:56:51.073685 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:51.073525 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bc6jp" podUID="cd2d6392-5d3a-49f6-be6a-954d6aaffba6" Aug 04 15:56:51.073685 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:51.073535 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zlj" Aug 04 15:56:51.073685 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:51.073591 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcb4z" Aug 04 15:56:51.073685 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:51.073652 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2zlj" podUID="e05ec517-319e-4360-8a0f-6133fb238192" Aug 04 15:56:51.073900 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:51.073744 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcb4z" podUID="fb454294-c3a4-4d8d-b705-95350ed95c13" Aug 04 15:56:53.073346 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:53.073309 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc6jp" Aug 04 15:56:53.073827 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:53.073432 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcb4z" Aug 04 15:56:53.073827 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:53.073443 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bc6jp" podUID="cd2d6392-5d3a-49f6-be6a-954d6aaffba6" Aug 04 15:56:53.073827 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:53.073474 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zlj" Aug 04 15:56:53.073827 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:53.073592 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcb4z" podUID="fb454294-c3a4-4d8d-b705-95350ed95c13" Aug 04 15:56:53.073827 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:53.073672 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2zlj" podUID="e05ec517-319e-4360-8a0f-6133fb238192" Aug 04 15:56:55.073144 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:55.073107 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcb4z" Aug 04 15:56:55.073605 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:55.073252 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcb4z" podUID="fb454294-c3a4-4d8d-b705-95350ed95c13" Aug 04 15:56:55.073605 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:55.073107 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zlj" Aug 04 15:56:55.073605 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:55.073107 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc6jp" Aug 04 15:56:55.073779 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:55.073747 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2zlj" podUID="e05ec517-319e-4360-8a0f-6133fb238192" Aug 04 15:56:55.073855 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:55.073833 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bc6jp" podUID="cd2d6392-5d3a-49f6-be6a-954d6aaffba6" Aug 04 15:56:57.072920 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:57.072888 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc6jp" Aug 04 15:56:57.072920 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:57.072905 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcb4z" Aug 04 15:56:57.073506 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:57.072886 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zlj" Aug 04 15:56:57.073506 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:57.073030 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bc6jp" podUID="cd2d6392-5d3a-49f6-be6a-954d6aaffba6" Aug 04 15:56:57.073506 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:57.073115 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2zlj" podUID="e05ec517-319e-4360-8a0f-6133fb238192" Aug 04 15:56:57.073506 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:57.073204 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcb4z" podUID="fb454294-c3a4-4d8d-b705-95350ed95c13" Aug 04 15:56:58.767680 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:58.767644 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bmm5\" (UniqueName: \"kubernetes.io/projected/cd2d6392-5d3a-49f6-be6a-954d6aaffba6-kube-api-access-6bmm5\") pod \"network-check-target-bc6jp\" (UID: \"cd2d6392-5d3a-49f6-be6a-954d6aaffba6\") " pod="openshift-network-diagnostics/network-check-target-bc6jp" Aug 04 15:56:58.767680 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:58.767684 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e05ec517-319e-4360-8a0f-6133fb238192-original-pull-secret\") pod \"global-pull-secret-syncer-q2zlj\" (UID: \"e05ec517-319e-4360-8a0f-6133fb238192\") " pod="kube-system/global-pull-secret-syncer-q2zlj" Aug 04 15:56:58.768187 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:58.767739 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb454294-c3a4-4d8d-b705-95350ed95c13-metrics-certs\") pod \"network-metrics-daemon-rcb4z\" (UID: \"fb454294-c3a4-4d8d-b705-95350ed95c13\") " pod="openshift-multus/network-metrics-daemon-rcb4z" Aug 04 15:56:58.768187 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:58.767847 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 15:56:58.768187 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:58.767873 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 15:56:58.768187 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:58.767873 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:56:58.768187 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:58.767887 2573 projected.go:196] Error preparing data for projected volume kube-api-access-6bmm5 for pod openshift-network-diagnostics/network-check-target-bc6jp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:56:58.768187 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:58.767873 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 15:56:58.768187 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:58.767938 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb454294-c3a4-4d8d-b705-95350ed95c13-metrics-certs podName:fb454294-c3a4-4d8d-b705-95350ed95c13 nodeName:}" failed. No retries permitted until 2026-08-04 15:57:14.767918264 +0000 UTC m=+34.185460372 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb454294-c3a4-4d8d-b705-95350ed95c13-metrics-certs") pod "network-metrics-daemon-rcb4z" (UID: "fb454294-c3a4-4d8d-b705-95350ed95c13") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:56:58.768187 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:58.767955 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd2d6392-5d3a-49f6-be6a-954d6aaffba6-kube-api-access-6bmm5 podName:cd2d6392-5d3a-49f6-be6a-954d6aaffba6 nodeName:}" failed. No retries permitted until 2026-08-04 15:57:14.767946154 +0000 UTC m=+34.185488261 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6bmm5" (UniqueName: "kubernetes.io/projected/cd2d6392-5d3a-49f6-be6a-954d6aaffba6-kube-api-access-6bmm5") pod "network-check-target-bc6jp" (UID: "cd2d6392-5d3a-49f6-be6a-954d6aaffba6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:56:58.768187 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:58.767974 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e05ec517-319e-4360-8a0f-6133fb238192-original-pull-secret podName:e05ec517-319e-4360-8a0f-6133fb238192 nodeName:}" failed. No retries permitted until 2026-08-04 15:57:14.767963823 +0000 UTC m=+34.185505932 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e05ec517-319e-4360-8a0f-6133fb238192-original-pull-secret") pod "global-pull-secret-syncer-q2zlj" (UID: "e05ec517-319e-4360-8a0f-6133fb238192") : object "kube-system"/"original-pull-secret" not registered Aug 04 15:56:59.073403 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:59.073323 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcb4z" Aug 04 15:56:59.073403 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:59.073355 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc6jp" Aug 04 15:56:59.073606 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:56:59.073323 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zlj" Aug 04 15:56:59.073606 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:59.073470 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcb4z" podUID="fb454294-c3a4-4d8d-b705-95350ed95c13" Aug 04 15:56:59.073706 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:59.073606 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2zlj" podUID="e05ec517-319e-4360-8a0f-6133fb238192" Aug 04 15:56:59.073706 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:56:59.073697 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bc6jp" podUID="cd2d6392-5d3a-49f6-be6a-954d6aaffba6" Aug 04 15:57:01.073697 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:01.073660 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zlj" Aug 04 15:57:01.074104 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:01.073769 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2zlj" podUID="e05ec517-319e-4360-8a0f-6133fb238192" Aug 04 15:57:01.074104 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:01.073794 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcb4z" Aug 04 15:57:01.074104 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:01.073816 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc6jp" Aug 04 15:57:01.074104 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:01.073889 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcb4z" podUID="fb454294-c3a4-4d8d-b705-95350ed95c13" Aug 04 15:57:01.074104 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:01.073957 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bc6jp" podUID="cd2d6392-5d3a-49f6-be6a-954d6aaffba6" Aug 04 15:57:02.204074 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:02.204026 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-210.ec2.internal" event={"ID":"fd26de1bd1ac6d99210ec0097a02c100","Type":"ContainerStarted","Data":"6a369880f62772c00dc1fcfcb09b413793edbff67a058046158d0a0aafbf9ccb"} Aug 04 15:57:02.206306 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:02.205940 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8jnqc" event={"ID":"975dcd50-33ff-4dbe-b7a8-43e44e042323","Type":"ContainerStarted","Data":"4dd930221f91223be99292c783072569a811c56b2fa1c209a81335bfbfa0d3bb"} Aug 04 15:57:02.207917 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:02.207889 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zfqf9" event={"ID":"482db4e4-6c5f-4c1e-9666-88f470fe67db","Type":"ContainerStarted","Data":"c9bcb98d82396772106723a60fdc8b724dbe4b68bcb15c2612713e5615dc7029"} Aug 04 15:57:02.210795 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:02.210769 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgqfs" event={"ID":"67933315-d76a-4b2c-8c92-f0249e607072","Type":"ContainerStarted","Data":"e4b0d84b2d5aa9745f2b56ddb20e9b8910e3558912e3f0dbddaa42fc5ac81f3e"} Aug 04 15:57:02.219700 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:02.219283 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-210.ec2.internal" podStartSLOduration=20.219266127 podStartE2EDuration="20.219266127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 15:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:57:02.219177561 +0000 UTC m=+21.636719688" watchObservedRunningTime="2026-08-04 15:57:02.219266127 +0000 UTC m=+21.636808255" Aug 04 15:57:02.219900 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:02.219856 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-210.ec2.internal" podStartSLOduration=20.219845706 podStartE2EDuration="20.219845706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 15:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:56:45.185672468 +0000 UTC m=+4.603214593" watchObservedRunningTime="2026-08-04 15:57:02.219845706 +0000 UTC m=+21.637387833" Aug 04 15:57:02.279498 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:02.279220 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8jnqc" podStartSLOduration=12.088963921 podStartE2EDuration="22.279197912s" totalImagesPullingTime="10.190233991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:56:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:57:02.263606995 +0000 UTC m=+21.681149119" watchObservedRunningTime="2026-08-04 15:57:02.279197912 +0000 UTC m=+21.696740039" Aug 04 15:57:02.279717 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:02.279673 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zfqf9" podStartSLOduration=3.5577882819999997 podStartE2EDuration="22.279656684s" totalImagesPullingTime="18.721868402s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 15:56:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:57:02.278847525 +0000 UTC m=+21.696389653" watchObservedRunningTime="2026-08-04 15:57:02.279656684 +0000 UTC m=+21.697198811" Aug 04 15:57:03.072751 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:03.072723 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcb4z" Aug 04 15:57:03.072952 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:03.072723 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zlj" Aug 04 15:57:03.072952 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:03.072830 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcb4z" podUID="fb454294-c3a4-4d8d-b705-95350ed95c13" Aug 04 15:57:03.072952 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:03.072869 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc6jp" Aug 04 15:57:03.073073 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:03.072952 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2zlj" podUID="e05ec517-319e-4360-8a0f-6133fb238192" Aug 04 15:57:03.075666 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:03.073362 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bc6jp" podUID="cd2d6392-5d3a-49f6-be6a-954d6aaffba6" Aug 04 15:57:03.215245 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:03.215202 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zcx94" event={"ID":"ef05b1ee-7f9a-42a0-825d-55ba18709e5f","Type":"ContainerStarted","Data":"4309f0bd9f67db02218565495c522e5dce160661e8de109b86121ba840991826"} Aug 04 15:57:03.216975 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:03.216943 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rz25w" event={"ID":"c10e0ff5-2215-4032-976a-907e6dccb2d3","Type":"ContainerStarted","Data":"9370ef3ba532e91b44422785c985792fc22190649ef44a59348cf20e44256630"} Aug 04 15:57:03.221359 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:03.221341 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mtbc5_11f9cbee-a773-420b-9873-5c8cb7679dd8/ovn-acl-logging/0.log" Aug 04 15:57:03.221671 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:03.221651 2573 generic.go:356] "Generic (PLEG): container finished" podID="11f9cbee-a773-420b-9873-5c8cb7679dd8" containerID="02faa0b35e53331996aa380b4db49ce936c7176724446ad5ad7120c50fd6f2fd" exitCode=1 Aug 04 15:57:03.221746 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:03.221721 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" event={"ID":"11f9cbee-a773-420b-9873-5c8cb7679dd8","Type":"ContainerStarted","Data":"43b5db6e6fd227c87db43f5dfa134a287fde501b50b906cef76ed4b1cead98f8"} Aug 04 15:57:03.221796 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:03.221779 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" event={"ID":"11f9cbee-a773-420b-9873-5c8cb7679dd8","Type":"ContainerStarted","Data":"c1a756f4bd5de8d0d259a7e3937fc623ddd492b4c82774aaca679b097c31ba42"} Aug 04 15:57:03.221841 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:03.221818 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" event={"ID":"11f9cbee-a773-420b-9873-5c8cb7679dd8","Type":"ContainerStarted","Data":"6137d6ce0608cf33a37080816a3351a199eaaac2a96441be5dc9e34823257fec"} Aug 04 15:57:03.221895 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:03.221848 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" event={"ID":"11f9cbee-a773-420b-9873-5c8cb7679dd8","Type":"ContainerStarted","Data":"5fde7befd504f78f54dcb8d4aa44418bf68ee2a6d9e5087f911e05368748ee80"} Aug 04 15:57:03.221895 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:03.221875 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" event={"ID":"11f9cbee-a773-420b-9873-5c8cb7679dd8","Type":"ContainerDied","Data":"02faa0b35e53331996aa380b4db49ce936c7176724446ad5ad7120c50fd6f2fd"} Aug 04 15:57:03.221994 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:03.221913 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" event={"ID":"11f9cbee-a773-420b-9873-5c8cb7679dd8","Type":"ContainerStarted","Data":"3a2ac99b6d9e45c94888c53d73dc6c47ee7d9ab9e425474dbbf9516797ff14a2"} Aug 04 15:57:03.223183 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:03.223158 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jbkjx" event={"ID":"abba07c5-995b-4a73-8a11-cde3a8326227","Type":"ContainerStarted","Data":"6edeb05a95054d5823484c13b148588db65ab18070dec4675f35fb90ddbd1eed"} Aug 04 15:57:03.224565 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:03.224528 2573 generic.go:356] "Generic (PLEG): container finished" podID="67933315-d76a-4b2c-8c92-f0249e607072" containerID="e4b0d84b2d5aa9745f2b56ddb20e9b8910e3558912e3f0dbddaa42fc5ac81f3e" exitCode=0 Aug 04 15:57:03.224662 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:03.224585 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgqfs" event={"ID":"67933315-d76a-4b2c-8c92-f0249e607072","Type":"ContainerDied","Data":"e4b0d84b2d5aa9745f2b56ddb20e9b8910e3558912e3f0dbddaa42fc5ac81f3e"} Aug 04 15:57:03.226039 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:03.226007 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lccb2" event={"ID":"588d2847-8d2d-4f62-97f0-2253584d2e07","Type":"ContainerStarted","Data":"0fdc8c9d8a0179813ec773e8c0a37ec2c840c9edf5e6a9b80cbd5e7b6addced9"} Aug 04 15:57:03.270215 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:03.270162 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zcx94" podStartSLOduration=4.562075254 podStartE2EDuration="23.270146359s" totalImagesPullingTime="18.708071105s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 15:56:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:57:03.239306169 +0000 UTC m=+22.656848305" watchObservedRunningTime="2026-08-04 15:57:03.270146359 +0000 UTC m=+22.687688485" Aug 04 15:57:03.330226 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:03.330146 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rz25w" podStartSLOduration=4.619671056 podStartE2EDuration="23.330108706s" totalImagesPullingTime="18.71043765s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 15:56:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:57:03.329671615 +0000 UTC m=+22.747213742" watchObservedRunningTime="2026-08-04 15:57:03.330108706 +0000 UTC m=+22.747650832" Aug 04 15:57:03.330414 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:03.330381 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jbkjx" podStartSLOduration=4.356909756 podStartE2EDuration="23.330371418s" totalImagesPullingTime="18.973461662s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 15:56:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:57:03.308333465 +0000 UTC m=+22.725875602" watchObservedRunningTime="2026-08-04 15:57:03.330371418 +0000 UTC m=+22.747913545" Aug 04 15:57:03.392055 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:03.392030 2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Aug 04 15:57:04.023184 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:04.023045 2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-04T15:57:03.392051551Z","UUID":"1e9059e7-e759-42db-bb72-8d6ec45106a8","Handler":null,"Name":"","Endpoint":""} Aug 04 15:57:04.025122 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:04.025094 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Aug 04 15:57:04.025262 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:04.025138 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Aug 04 15:57:04.228907 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:04.228873 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kg84j" event={"ID":"3932e5e3-69c2-4cee-ad2b-ac496485b140","Type":"ContainerStarted","Data":"9a4bfb20e2cb141371aa5a5b0f02237f12d534cb992bc0bbfad1e4b43f8c4b26"} Aug 04 15:57:04.230832 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:04.230800 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lccb2" event={"ID":"588d2847-8d2d-4f62-97f0-2253584d2e07","Type":"ContainerStarted","Data":"1d4baf2ec45420861f54c4d017fe4faa2c1468204f1d48ea19aa4e7ef73f1840"} Aug 04 15:57:04.245004 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:04.244962 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kg84j" podStartSLOduration=5.516816759 podStartE2EDuration="24.244947862s" totalImagesPullingTime="18.728131103s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 15:56:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:57:04.24465427 +0000 UTC m=+23.662196395" watchObservedRunningTime="2026-08-04 15:57:04.244947862 +0000 UTC m=+23.662489987" Aug 04 15:57:05.072999 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:05.072908 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc6jp" Aug 04 15:57:05.073155 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:05.072919 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zlj" Aug 04 15:57:05.073155 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:05.073041 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bc6jp" podUID="cd2d6392-5d3a-49f6-be6a-954d6aaffba6" Aug 04 15:57:05.073155 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:05.073065 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcb4z" Aug 04 15:57:05.073351 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:05.073160 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2zlj" podUID="e05ec517-319e-4360-8a0f-6133fb238192" Aug 04 15:57:05.073351 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:05.073270 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcb4z" podUID="fb454294-c3a4-4d8d-b705-95350ed95c13" Aug 04 15:57:05.235830 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:05.235804 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mtbc5_11f9cbee-a773-420b-9873-5c8cb7679dd8/ovn-acl-logging/0.log" Aug 04 15:57:05.236257 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:05.236236 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" event={"ID":"11f9cbee-a773-420b-9873-5c8cb7679dd8","Type":"ContainerStarted","Data":"7c703ebcf070f4e461a0240829e036e2500637a08175697c9f2552c8a8a536a1"} Aug 04 15:57:05.692995 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:05.692735 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8jnqc" Aug 04 15:57:05.693502 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:05.693472 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8jnqc" Aug 04 15:57:06.238113 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:06.238078 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8jnqc" Aug 04 15:57:06.238672 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:06.238655 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8jnqc" Aug 04 15:57:07.073409 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:07.073283 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc6jp" Aug 04 15:57:07.073409 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:07.073323 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zlj" Aug 04 15:57:07.073409 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:07.073334 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcb4z" Aug 04 15:57:07.073687 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:07.073433 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bc6jp" podUID="cd2d6392-5d3a-49f6-be6a-954d6aaffba6" Aug 04 15:57:07.073687 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:07.073524 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2zlj" podUID="e05ec517-319e-4360-8a0f-6133fb238192" Aug 04 15:57:07.073687 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:07.073659 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcb4z" podUID="fb454294-c3a4-4d8d-b705-95350ed95c13" Aug 04 15:57:07.242192 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:07.242158 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lccb2" event={"ID":"588d2847-8d2d-4f62-97f0-2253584d2e07","Type":"ContainerStarted","Data":"b1923861dd569ae3f8f0a2f9e37c4f2de4f492f618e73f7e06739409d2e12f1f"} Aug 04 15:57:07.245963 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:07.245936 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mtbc5_11f9cbee-a773-420b-9873-5c8cb7679dd8/ovn-acl-logging/0.log" Aug 04 15:57:07.246483 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:07.246453 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" event={"ID":"11f9cbee-a773-420b-9873-5c8cb7679dd8","Type":"ContainerStarted","Data":"a0c0a158e1e0998d376a900043a77a1fbb92196c2bfbe74616243fd1ee4d94af"} Aug 04 15:57:07.246747 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:07.246713 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:57:07.246864 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:07.246787 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:57:07.246923 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:07.246906 2573 scope.go:117] "RemoveContainer" containerID="02faa0b35e53331996aa380b4db49ce936c7176724446ad5ad7120c50fd6f2fd" Aug 04 15:57:07.263202 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:07.263021 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lccb2" podStartSLOduration=4.603184077 podStartE2EDuration="27.263005341s" totalImagesPullingTime="22.659821264s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 15:56:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:57:07.262832576 +0000 UTC m=+26.680374704" watchObservedRunningTime="2026-08-04 15:57:07.263005341 +0000 UTC m=+26.680547467" Aug 04 15:57:07.263855 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:07.263832 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:57:09.073171 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:09.073089 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zlj" Aug 04 15:57:09.073638 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:09.073089 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc6jp" Aug 04 15:57:09.073638 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:09.073201 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2zlj" podUID="e05ec517-319e-4360-8a0f-6133fb238192" Aug 04 15:57:09.073638 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:09.073316 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bc6jp" podUID="cd2d6392-5d3a-49f6-be6a-954d6aaffba6" Aug 04 15:57:09.073638 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:09.073104 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcb4z" Aug 04 15:57:09.073638 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:09.073468 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcb4z" podUID="fb454294-c3a4-4d8d-b705-95350ed95c13" Aug 04 15:57:09.164376 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:09.164345 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q2zlj"] Aug 04 15:57:09.173845 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:09.173793 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rcb4z"] Aug 04 15:57:09.174374 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:09.174355 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bc6jp"] Aug 04 15:57:09.252992 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:09.252957 2573 generic.go:356] "Generic (PLEG): container finished" podID="67933315-d76a-4b2c-8c92-f0249e607072" containerID="fec1ce0d73e32197cd3c412c0893938f0f71f793234bdd71c027943151dc77d8" exitCode=0 Aug 04 15:57:09.253173 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:09.253051 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgqfs" event={"ID":"67933315-d76a-4b2c-8c92-f0249e607072","Type":"ContainerDied","Data":"fec1ce0d73e32197cd3c412c0893938f0f71f793234bdd71c027943151dc77d8"} Aug 04 15:57:09.257084 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:09.257065 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mtbc5_11f9cbee-a773-420b-9873-5c8cb7679dd8/ovn-acl-logging/0.log" Aug 04 15:57:09.257442 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:09.257426 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc6jp" Aug 04 15:57:09.257504 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:09.257455 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcb4z" Aug 04 15:57:09.257504 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:09.257423 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" event={"ID":"11f9cbee-a773-420b-9873-5c8cb7679dd8","Type":"ContainerStarted","Data":"eee5bd258d35e337873b4962771b37a62ee74051cfcb74d416e1f7b31f2f9452"} Aug 04 15:57:09.257615 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:09.257428 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zlj" Aug 04 15:57:09.257615 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:09.257523 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bc6jp" podUID="cd2d6392-5d3a-49f6-be6a-954d6aaffba6" Aug 04 15:57:09.257684 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:09.257616 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2zlj" podUID="e05ec517-319e-4360-8a0f-6133fb238192" Aug 04 15:57:09.257684 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:09.257662 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcb4z" podUID="fb454294-c3a4-4d8d-b705-95350ed95c13" Aug 04 15:57:09.257832 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:09.257815 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:57:09.273819 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:09.273795 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:57:11.073585 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:11.073370 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc6jp" Aug 04 15:57:11.074002 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:11.073443 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcb4z" Aug 04 15:57:11.074002 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:11.073661 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bc6jp" podUID="cd2d6392-5d3a-49f6-be6a-954d6aaffba6" Aug 04 15:57:11.074002 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:11.073463 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zlj" Aug 04 15:57:11.074002 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:11.073735 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcb4z" podUID="fb454294-c3a4-4d8d-b705-95350ed95c13" Aug 04 15:57:11.074002 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:11.073851 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2zlj" podUID="e05ec517-319e-4360-8a0f-6133fb238192" Aug 04 15:57:11.263944 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:11.263909 2573 generic.go:356] "Generic (PLEG): container finished" podID="67933315-d76a-4b2c-8c92-f0249e607072" containerID="28430e46bafc37173b3b7c98dd51c8fc9a96c715d2d7392aa2b2c4b58c72f8a5" exitCode=0 Aug 04 15:57:11.264118 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:11.263986 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgqfs" event={"ID":"67933315-d76a-4b2c-8c92-f0249e607072","Type":"ContainerDied","Data":"28430e46bafc37173b3b7c98dd51c8fc9a96c715d2d7392aa2b2c4b58c72f8a5"} Aug 04 15:57:11.284860 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:11.284818 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" podUID="11f9cbee-a773-420b-9873-5c8cb7679dd8" containerName="ovnkube-controller" probeResult="failure" output="" Aug 04 15:57:11.298003 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:11.297959 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" podStartSLOduration=12.542526625 podStartE2EDuration="31.297942056s" totalImagesPullingTime="18.755415431s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 15:56:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:57:09.325411214 +0000 UTC m=+28.742953377" watchObservedRunningTime="2026-08-04 15:57:11.297942056 +0000 UTC m=+30.715484160" Aug 04 15:57:13.073365 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:13.073323 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zlj" Aug 04 15:57:13.074035 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:13.073432 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcb4z" Aug 04 15:57:13.074035 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:13.073439 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2zlj" podUID="e05ec517-319e-4360-8a0f-6133fb238192" Aug 04 15:57:13.074035 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:13.073463 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc6jp" Aug 04 15:57:13.074035 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:13.073523 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rcb4z" podUID="fb454294-c3a4-4d8d-b705-95350ed95c13" Aug 04 15:57:13.074035 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:13.073616 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bc6jp" podUID="cd2d6392-5d3a-49f6-be6a-954d6aaffba6" Aug 04 15:57:13.271157 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:13.271127 2573 generic.go:356] "Generic (PLEG): container finished" podID="67933315-d76a-4b2c-8c92-f0249e607072" containerID="bc89bcd8c4771ba3c0f956c92f5704d0de7fd795ea602e4f87dd8de387a6589c" exitCode=0 Aug 04 15:57:13.271298 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:13.271164 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgqfs" event={"ID":"67933315-d76a-4b2c-8c92-f0249e607072","Type":"ContainerDied","Data":"bc89bcd8c4771ba3c0f956c92f5704d0de7fd795ea602e4f87dd8de387a6589c"} Aug 04 15:57:14.776391 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:14.776364 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e05ec517-319e-4360-8a0f-6133fb238192-original-pull-secret\") pod \"global-pull-secret-syncer-q2zlj\" (UID: \"e05ec517-319e-4360-8a0f-6133fb238192\") " pod="kube-system/global-pull-secret-syncer-q2zlj" Aug 04 15:57:14.776782 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:14.776422 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb454294-c3a4-4d8d-b705-95350ed95c13-metrics-certs\") pod \"network-metrics-daemon-rcb4z\" (UID: \"fb454294-c3a4-4d8d-b705-95350ed95c13\") " pod="openshift-multus/network-metrics-daemon-rcb4z" Aug 04 15:57:14.776782 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:14.776462 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bmm5\" (UniqueName: \"kubernetes.io/projected/cd2d6392-5d3a-49f6-be6a-954d6aaffba6-kube-api-access-6bmm5\") pod \"network-check-target-bc6jp\" (UID: \"cd2d6392-5d3a-49f6-be6a-954d6aaffba6\") " pod="openshift-network-diagnostics/network-check-target-bc6jp" Aug 04 15:57:14.776782 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:14.776519 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 15:57:14.776782 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:14.776573 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:57:14.776782 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:14.776591 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 15:57:14.776782 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:14.776604 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e05ec517-319e-4360-8a0f-6133fb238192-original-pull-secret podName:e05ec517-319e-4360-8a0f-6133fb238192 nodeName:}" failed. No retries permitted until 2026-08-04 15:57:46.776588182 +0000 UTC m=+66.194130289 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e05ec517-319e-4360-8a0f-6133fb238192-original-pull-secret") pod "global-pull-secret-syncer-q2zlj" (UID: "e05ec517-319e-4360-8a0f-6133fb238192") : object "kube-system"/"original-pull-secret" not registered Aug 04 15:57:14.776782 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:14.776620 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb454294-c3a4-4d8d-b705-95350ed95c13-metrics-certs podName:fb454294-c3a4-4d8d-b705-95350ed95c13 nodeName:}" failed. No retries permitted until 2026-08-04 15:57:46.776613192 +0000 UTC m=+66.194155296 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb454294-c3a4-4d8d-b705-95350ed95c13-metrics-certs") pod "network-metrics-daemon-rcb4z" (UID: "fb454294-c3a4-4d8d-b705-95350ed95c13") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:57:14.776782 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:14.776606 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 15:57:14.776782 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:14.776633 2573 projected.go:196] Error preparing data for projected volume kube-api-access-6bmm5 for pod openshift-network-diagnostics/network-check-target-bc6jp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:57:14.776782 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:14.776657 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd2d6392-5d3a-49f6-be6a-954d6aaffba6-kube-api-access-6bmm5 podName:cd2d6392-5d3a-49f6-be6a-954d6aaffba6 nodeName:}" failed. No retries permitted until 2026-08-04 15:57:46.776652089 +0000 UTC m=+66.194194192 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6bmm5" (UniqueName: "kubernetes.io/projected/cd2d6392-5d3a-49f6-be6a-954d6aaffba6-kube-api-access-6bmm5") pod "network-check-target-bc6jp" (UID: "cd2d6392-5d3a-49f6-be6a-954d6aaffba6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:57:14.849831 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:14.849803 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-210.ec2.internal" event="NodeReady" Aug 04 15:57:14.850014 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:14.849972 2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Aug 04 15:57:14.901698 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:14.901450 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jpd9h"] Aug 04 15:57:14.916521 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:14.916492 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-q6m46"] Aug 04 15:57:14.916705 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:14.916648 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jpd9h" Aug 04 15:57:14.920379 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:14.920350 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Aug 04 15:57:14.920983 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:14.920714 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fnzc4\"" Aug 04 15:57:14.920983 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:14.920875 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Aug 04 15:57:14.925861 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:14.925841 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-q6m46" Aug 04 15:57:14.926253 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:14.925988 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jpd9h"] Aug 04 15:57:14.926538 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:14.926519 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-q6m46"] Aug 04 15:57:14.927883 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:14.927862 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Aug 04 15:57:14.927991 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:14.927937 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Aug 04 15:57:14.928136 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:14.928119 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Aug 04 15:57:14.928315 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:14.928292 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-752h4\"" Aug 04 15:57:14.977735 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:14.977694 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68c83082-033c-4f0a-83ad-6291422eac03-tmp-dir\") pod \"dns-default-jpd9h\" (UID: \"68c83082-033c-4f0a-83ad-6291422eac03\") " pod="openshift-dns/dns-default-jpd9h" Aug 04 15:57:14.977913 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:14.977753 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80648f1b-68e3-44d7-b0f9-8e93d70c9413-cert\") pod \"ingress-canary-q6m46\" (UID: \"80648f1b-68e3-44d7-b0f9-8e93d70c9413\") " pod="openshift-ingress-canary/ingress-canary-q6m46" Aug 04 15:57:14.977913 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:14.977783 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/68c83082-033c-4f0a-83ad-6291422eac03-config-volume\") pod \"dns-default-jpd9h\" (UID: \"68c83082-033c-4f0a-83ad-6291422eac03\") " pod="openshift-dns/dns-default-jpd9h" Aug 04 15:57:14.977913 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:14.977893 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncnhx\" (UniqueName: \"kubernetes.io/projected/80648f1b-68e3-44d7-b0f9-8e93d70c9413-kube-api-access-ncnhx\") pod \"ingress-canary-q6m46\" (UID: \"80648f1b-68e3-44d7-b0f9-8e93d70c9413\") " pod="openshift-ingress-canary/ingress-canary-q6m46" Aug 04 15:57:14.978051 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:14.977922 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5sjp\" (UniqueName: \"kubernetes.io/projected/68c83082-033c-4f0a-83ad-6291422eac03-kube-api-access-l5sjp\") pod \"dns-default-jpd9h\" (UID: \"68c83082-033c-4f0a-83ad-6291422eac03\") " pod="openshift-dns/dns-default-jpd9h" Aug 04 15:57:14.978051 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:14.977973 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68c83082-033c-4f0a-83ad-6291422eac03-metrics-tls\") pod \"dns-default-jpd9h\" (UID: \"68c83082-033c-4f0a-83ad-6291422eac03\") " pod="openshift-dns/dns-default-jpd9h" Aug 04 15:57:15.073412 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:15.073330 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zlj" Aug 04 15:57:15.073626 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:15.073335 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc6jp" Aug 04 15:57:15.073626 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:15.073339 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcb4z" Aug 04 15:57:15.076127 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:15.076102 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bfprq\"" Aug 04 15:57:15.076266 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:15.076180 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xj8nj\"" Aug 04 15:57:15.076266 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:15.076106 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 15:57:15.076266 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:15.076257 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 04 15:57:15.076429 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:15.076266 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 15:57:15.076429 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:15.076102 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 04 15:57:15.078311 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:15.078287 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68c83082-033c-4f0a-83ad-6291422eac03-tmp-dir\") pod \"dns-default-jpd9h\" (UID: \"68c83082-033c-4f0a-83ad-6291422eac03\") " pod="openshift-dns/dns-default-jpd9h" Aug 04 15:57:15.078823 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:15.078714 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68c83082-033c-4f0a-83ad-6291422eac03-tmp-dir\") pod \"dns-default-jpd9h\" (UID: \"68c83082-033c-4f0a-83ad-6291422eac03\") " pod="openshift-dns/dns-default-jpd9h" Aug 04 15:57:15.079043 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:15.079024 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80648f1b-68e3-44d7-b0f9-8e93d70c9413-cert\") pod \"ingress-canary-q6m46\" (UID: \"80648f1b-68e3-44d7-b0f9-8e93d70c9413\") " pod="openshift-ingress-canary/ingress-canary-q6m46" Aug 04 15:57:15.079131 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:15.079065 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/68c83082-033c-4f0a-83ad-6291422eac03-config-volume\") pod \"dns-default-jpd9h\" (UID: \"68c83082-033c-4f0a-83ad-6291422eac03\") " pod="openshift-dns/dns-default-jpd9h" Aug 04 15:57:15.079193 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:15.079148 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncnhx\" (UniqueName: \"kubernetes.io/projected/80648f1b-68e3-44d7-b0f9-8e93d70c9413-kube-api-access-ncnhx\") pod \"ingress-canary-q6m46\" (UID: \"80648f1b-68e3-44d7-b0f9-8e93d70c9413\") " pod="openshift-ingress-canary/ingress-canary-q6m46" Aug 04 15:57:15.079193 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:15.079182 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5sjp\" (UniqueName: \"kubernetes.io/projected/68c83082-033c-4f0a-83ad-6291422eac03-kube-api-access-l5sjp\") pod \"dns-default-jpd9h\" (UID: \"68c83082-033c-4f0a-83ad-6291422eac03\") " pod="openshift-dns/dns-default-jpd9h" Aug 04 15:57:15.079293 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:15.079209 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68c83082-033c-4f0a-83ad-6291422eac03-metrics-tls\") pod \"dns-default-jpd9h\" (UID: \"68c83082-033c-4f0a-83ad-6291422eac03\") " pod="openshift-dns/dns-default-jpd9h" Aug 04 15:57:15.079344 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:15.079328 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 15:57:15.079396 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:15.079392 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68c83082-033c-4f0a-83ad-6291422eac03-metrics-tls podName:68c83082-033c-4f0a-83ad-6291422eac03 nodeName:}" failed. No retries permitted until 2026-08-04 15:57:15.579371681 +0000 UTC m=+34.996913801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68c83082-033c-4f0a-83ad-6291422eac03-metrics-tls") pod "dns-default-jpd9h" (UID: "68c83082-033c-4f0a-83ad-6291422eac03") : secret "dns-default-metrics-tls" not found Aug 04 15:57:15.079991 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:15.079900 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/68c83082-033c-4f0a-83ad-6291422eac03-config-volume\") pod \"dns-default-jpd9h\" (UID: \"68c83082-033c-4f0a-83ad-6291422eac03\") " pod="openshift-dns/dns-default-jpd9h" Aug 04 15:57:15.084961 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:15.084936 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80648f1b-68e3-44d7-b0f9-8e93d70c9413-cert\") pod \"ingress-canary-q6m46\" (UID: \"80648f1b-68e3-44d7-b0f9-8e93d70c9413\") " pod="openshift-ingress-canary/ingress-canary-q6m46" Aug 04 15:57:15.090483 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:15.090455 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncnhx\" (UniqueName: \"kubernetes.io/projected/80648f1b-68e3-44d7-b0f9-8e93d70c9413-kube-api-access-ncnhx\") pod \"ingress-canary-q6m46\" (UID: \"80648f1b-68e3-44d7-b0f9-8e93d70c9413\") " pod="openshift-ingress-canary/ingress-canary-q6m46" Aug 04 15:57:15.092813 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:15.092791 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5sjp\" (UniqueName: \"kubernetes.io/projected/68c83082-033c-4f0a-83ad-6291422eac03-kube-api-access-l5sjp\") pod \"dns-default-jpd9h\" (UID: \"68c83082-033c-4f0a-83ad-6291422eac03\") " pod="openshift-dns/dns-default-jpd9h" Aug 04 15:57:15.236947 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:15.236911 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-q6m46" Aug 04 15:57:15.438256 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:15.437945 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-q6m46"] Aug 04 15:57:15.442677 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:57:15.442646 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80648f1b_68e3_44d7_b0f9_8e93d70c9413.slice/crio-ad4cce790f68b34543b4f604b758f589c6e5567ac96f15eb691623c4c6055f2b WatchSource:0}: Error finding container ad4cce790f68b34543b4f604b758f589c6e5567ac96f15eb691623c4c6055f2b: Status 404 returned error can't find the container with id ad4cce790f68b34543b4f604b758f589c6e5567ac96f15eb691623c4c6055f2b Aug 04 15:57:15.582707 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:15.582670 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68c83082-033c-4f0a-83ad-6291422eac03-metrics-tls\") pod \"dns-default-jpd9h\" (UID: \"68c83082-033c-4f0a-83ad-6291422eac03\") " pod="openshift-dns/dns-default-jpd9h" Aug 04 15:57:15.582885 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:15.582780 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 15:57:15.582885 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:15.582838 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68c83082-033c-4f0a-83ad-6291422eac03-metrics-tls podName:68c83082-033c-4f0a-83ad-6291422eac03 nodeName:}" failed. No retries permitted until 2026-08-04 15:57:16.582825 +0000 UTC m=+36.000367105 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68c83082-033c-4f0a-83ad-6291422eac03-metrics-tls") pod "dns-default-jpd9h" (UID: "68c83082-033c-4f0a-83ad-6291422eac03") : secret "dns-default-metrics-tls" not found Aug 04 15:57:16.282467 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:16.282425 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-q6m46" event={"ID":"80648f1b-68e3-44d7-b0f9-8e93d70c9413","Type":"ContainerStarted","Data":"ad4cce790f68b34543b4f604b758f589c6e5567ac96f15eb691623c4c6055f2b"} Aug 04 15:57:16.592326 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:16.592231 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68c83082-033c-4f0a-83ad-6291422eac03-metrics-tls\") pod \"dns-default-jpd9h\" (UID: \"68c83082-033c-4f0a-83ad-6291422eac03\") " pod="openshift-dns/dns-default-jpd9h" Aug 04 15:57:16.592504 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:16.592390 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 15:57:16.592504 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:16.592459 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68c83082-033c-4f0a-83ad-6291422eac03-metrics-tls podName:68c83082-033c-4f0a-83ad-6291422eac03 nodeName:}" failed. No retries permitted until 2026-08-04 15:57:18.592440238 +0000 UTC m=+38.009982345 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68c83082-033c-4f0a-83ad-6291422eac03-metrics-tls") pod "dns-default-jpd9h" (UID: "68c83082-033c-4f0a-83ad-6291422eac03") : secret "dns-default-metrics-tls" not found Aug 04 15:57:18.605923 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:18.605881 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68c83082-033c-4f0a-83ad-6291422eac03-metrics-tls\") pod \"dns-default-jpd9h\" (UID: \"68c83082-033c-4f0a-83ad-6291422eac03\") " pod="openshift-dns/dns-default-jpd9h" Aug 04 15:57:18.606285 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:18.606031 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 15:57:18.606285 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:18.606095 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68c83082-033c-4f0a-83ad-6291422eac03-metrics-tls podName:68c83082-033c-4f0a-83ad-6291422eac03 nodeName:}" failed. No retries permitted until 2026-08-04 15:57:22.606079409 +0000 UTC m=+42.023621513 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68c83082-033c-4f0a-83ad-6291422eac03-metrics-tls") pod "dns-default-jpd9h" (UID: "68c83082-033c-4f0a-83ad-6291422eac03") : secret "dns-default-metrics-tls" not found Aug 04 15:57:19.293984 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:19.293810 2573 generic.go:356] "Generic (PLEG): container finished" podID="67933315-d76a-4b2c-8c92-f0249e607072" containerID="2d16641cfc8bf15ce6d86e77957aea5d080aa93570718f26958397c97d27c31f" exitCode=0 Aug 04 15:57:19.294158 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:19.293889 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgqfs" event={"ID":"67933315-d76a-4b2c-8c92-f0249e607072","Type":"ContainerDied","Data":"2d16641cfc8bf15ce6d86e77957aea5d080aa93570718f26958397c97d27c31f"} Aug 04 15:57:19.295384 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:19.295364 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-q6m46" event={"ID":"80648f1b-68e3-44d7-b0f9-8e93d70c9413","Type":"ContainerStarted","Data":"ab8eb893abb1ffb8d62d3d22c45674bd21683b37ba85e1c6912ed1453acfebaa"} Aug 04 15:57:19.337239 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:19.337182 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-q6m46" podStartSLOduration=2.011822884 podStartE2EDuration="5.337159743s" totalImagesPullingTime="3.325336859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:57:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:57:19.336891149 +0000 UTC m=+38.754433280" watchObservedRunningTime="2026-08-04 15:57:19.337159743 +0000 UTC m=+38.754701864" Aug 04 15:57:20.301191 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:20.301163 2573 generic.go:356] "Generic (PLEG): container finished" podID="67933315-d76a-4b2c-8c92-f0249e607072" containerID="58b12be1acd3d76b3e09ef2083f35e6c707cce8982ffc2655f47fd9d807d96a0" exitCode=0 Aug 04 15:57:20.301767 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:20.301251 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgqfs" event={"ID":"67933315-d76a-4b2c-8c92-f0249e607072","Type":"ContainerDied","Data":"58b12be1acd3d76b3e09ef2083f35e6c707cce8982ffc2655f47fd9d807d96a0"} Aug 04 15:57:21.306990 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:21.306955 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgqfs" event={"ID":"67933315-d76a-4b2c-8c92-f0249e607072","Type":"ContainerStarted","Data":"208b8889d15d3e50f8773a031b845ee6759b35f399a5ac99efe6b784b331d090"} Aug 04 15:57:21.336735 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:21.336687 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bgqfs" podStartSLOduration=9.139440663 podStartE2EDuration="41.336671933s" totalImagesPullingTime="32.19723127s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 15:56:40 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:57:21.335308766 +0000 UTC m=+40.752850892" watchObservedRunningTime="2026-08-04 15:57:21.336671933 +0000 UTC m=+40.754214059" Aug 04 15:57:22.633027 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:22.632988 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68c83082-033c-4f0a-83ad-6291422eac03-metrics-tls\") pod \"dns-default-jpd9h\" (UID: \"68c83082-033c-4f0a-83ad-6291422eac03\") " pod="openshift-dns/dns-default-jpd9h" Aug 04 15:57:22.633415 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:22.633100 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 15:57:22.633415 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:22.633150 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68c83082-033c-4f0a-83ad-6291422eac03-metrics-tls podName:68c83082-033c-4f0a-83ad-6291422eac03 nodeName:}" failed. No retries permitted until 2026-08-04 15:57:30.633136961 +0000 UTC m=+50.050679066 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68c83082-033c-4f0a-83ad-6291422eac03-metrics-tls") pod "dns-default-jpd9h" (UID: "68c83082-033c-4f0a-83ad-6291422eac03") : secret "dns-default-metrics-tls" not found Aug 04 15:57:24.869684 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:24.869650 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-gkbnm"] Aug 04 15:57:24.915125 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:24.915089 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7c65459b9d-v7hwq"] Aug 04 15:57:24.915268 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:24.915206 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-gkbnm" Aug 04 15:57:24.917456 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:24.917435 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-xpbr5\"" Aug 04 15:57:24.931158 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:24.931134 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7c65459b9d-v7hwq"] Aug 04 15:57:24.931277 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:24.931201 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-gkbnm"] Aug 04 15:57:24.931339 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:24.931308 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7c65459b9d-v7hwq" Aug 04 15:57:24.934227 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:24.934203 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Aug 04 15:57:24.934475 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:24.934459 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Aug 04 15:57:24.934649 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:24.934625 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Aug 04 15:57:24.934749 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:24.934692 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Aug 04 15:57:24.934796 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:24.934765 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-t65s4\"" Aug 04 15:57:24.934796 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:24.934767 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Aug 04 15:57:24.934796 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:24.934776 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Aug 04 15:57:24.971654 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:24.971621 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-d97mf"] Aug 04 15:57:24.994980 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:24.994954 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-gvb4l"] Aug 04 15:57:24.995144 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:24.995079 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-d97mf" Aug 04 15:57:24.997236 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:24.997215 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-x6t5w\"" Aug 04 15:57:24.997436 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:24.997421 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Aug 04 15:57:25.000792 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.000774 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Aug 04 15:57:25.018334 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.018308 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-d97mf"] Aug 04 15:57:25.018445 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.018386 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-gvb4l"] Aug 04 15:57:25.018445 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.018436 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-gvb4l" Aug 04 15:57:25.021261 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.021242 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Aug 04 15:57:25.021582 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.021314 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Aug 04 15:57:25.021582 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.021344 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-gjznf\"" Aug 04 15:57:25.021704 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.021675 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Aug 04 15:57:25.021704 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.021682 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Aug 04 15:57:25.028268 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.028251 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Aug 04 15:57:25.051316 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.051292 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97cpr\" (UniqueName: \"kubernetes.io/projected/820a26ee-edcd-4e18-9152-69027236d7a3-kube-api-access-97cpr\") pod \"router-default-7c65459b9d-v7hwq\" (UID: \"820a26ee-edcd-4e18-9152-69027236d7a3\") " pod="openshift-ingress/router-default-7c65459b9d-v7hwq" Aug 04 15:57:25.051455 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.051334 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/820a26ee-edcd-4e18-9152-69027236d7a3-default-certificate\") pod \"router-default-7c65459b9d-v7hwq\" (UID: \"820a26ee-edcd-4e18-9152-69027236d7a3\") " pod="openshift-ingress/router-default-7c65459b9d-v7hwq" Aug 04 15:57:25.051455 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.051382 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/820a26ee-edcd-4e18-9152-69027236d7a3-service-ca-bundle\") pod \"router-default-7c65459b9d-v7hwq\" (UID: \"820a26ee-edcd-4e18-9152-69027236d7a3\") " pod="openshift-ingress/router-default-7c65459b9d-v7hwq" Aug 04 15:57:25.051455 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.051431 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzxlz\" (UniqueName: \"kubernetes.io/projected/f715e6cb-4601-489a-9e05-af3fbd46d394-kube-api-access-dzxlz\") pod \"network-check-source-65bdbbff45-gkbnm\" (UID: \"f715e6cb-4601-489a-9e05-af3fbd46d394\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-gkbnm" Aug 04 15:57:25.051572 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.051461 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/820a26ee-edcd-4e18-9152-69027236d7a3-stats-auth\") pod \"router-default-7c65459b9d-v7hwq\" (UID: \"820a26ee-edcd-4e18-9152-69027236d7a3\") " pod="openshift-ingress/router-default-7c65459b9d-v7hwq" Aug 04 15:57:25.051572 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.051489 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/820a26ee-edcd-4e18-9152-69027236d7a3-metrics-certs\") pod \"router-default-7c65459b9d-v7hwq\" (UID: \"820a26ee-edcd-4e18-9152-69027236d7a3\") " pod="openshift-ingress/router-default-7c65459b9d-v7hwq" Aug 04 15:57:25.152848 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.152768 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/820a26ee-edcd-4e18-9152-69027236d7a3-default-certificate\") pod \"router-default-7c65459b9d-v7hwq\" (UID: \"820a26ee-edcd-4e18-9152-69027236d7a3\") " pod="openshift-ingress/router-default-7c65459b9d-v7hwq" Aug 04 15:57:25.152848 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.152817 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c16f82ab-8340-49ee-b41e-5685d4592fd2-serving-cert\") pod \"console-operator-5ccd6b445f-gvb4l\" (UID: \"c16f82ab-8340-49ee-b41e-5685d4592fd2\") " pod="openshift-console-operator/console-operator-5ccd6b445f-gvb4l" Aug 04 15:57:25.153044 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.152926 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvpfm\" (UniqueName: \"kubernetes.io/projected/c06a604b-4484-4ead-a88b-d64926af2a2b-kube-api-access-qvpfm\") pod \"volume-data-source-validator-5b8776669-d97mf\" (UID: \"c06a604b-4484-4ead-a88b-d64926af2a2b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-d97mf" Aug 04 15:57:25.153044 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.152975 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/820a26ee-edcd-4e18-9152-69027236d7a3-service-ca-bundle\") pod \"router-default-7c65459b9d-v7hwq\" (UID: \"820a26ee-edcd-4e18-9152-69027236d7a3\") " pod="openshift-ingress/router-default-7c65459b9d-v7hwq" Aug 04 15:57:25.153044 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.152999 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c16f82ab-8340-49ee-b41e-5685d4592fd2-config\") pod \"console-operator-5ccd6b445f-gvb4l\" (UID: \"c16f82ab-8340-49ee-b41e-5685d4592fd2\") " pod="openshift-console-operator/console-operator-5ccd6b445f-gvb4l" Aug 04 15:57:25.153044 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.153021 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzxlz\" (UniqueName: \"kubernetes.io/projected/f715e6cb-4601-489a-9e05-af3fbd46d394-kube-api-access-dzxlz\") pod \"network-check-source-65bdbbff45-gkbnm\" (UID: \"f715e6cb-4601-489a-9e05-af3fbd46d394\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-gkbnm" Aug 04 15:57:25.153044 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.153037 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/820a26ee-edcd-4e18-9152-69027236d7a3-stats-auth\") pod \"router-default-7c65459b9d-v7hwq\" (UID: \"820a26ee-edcd-4e18-9152-69027236d7a3\") " pod="openshift-ingress/router-default-7c65459b9d-v7hwq" Aug 04 15:57:25.153280 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.153060 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/820a26ee-edcd-4e18-9152-69027236d7a3-metrics-certs\") pod \"router-default-7c65459b9d-v7hwq\" (UID: \"820a26ee-edcd-4e18-9152-69027236d7a3\") " pod="openshift-ingress/router-default-7c65459b9d-v7hwq" Aug 04 15:57:25.153280 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:25.153078 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/820a26ee-edcd-4e18-9152-69027236d7a3-service-ca-bundle podName:820a26ee-edcd-4e18-9152-69027236d7a3 nodeName:}" failed. No retries permitted until 2026-08-04 15:57:25.653062478 +0000 UTC m=+45.070604582 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/820a26ee-edcd-4e18-9152-69027236d7a3-service-ca-bundle") pod "router-default-7c65459b9d-v7hwq" (UID: "820a26ee-edcd-4e18-9152-69027236d7a3") : configmap references non-existent config key: service-ca.crt Aug 04 15:57:25.153280 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.153099 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c16f82ab-8340-49ee-b41e-5685d4592fd2-trusted-ca\") pod \"console-operator-5ccd6b445f-gvb4l\" (UID: \"c16f82ab-8340-49ee-b41e-5685d4592fd2\") " pod="openshift-console-operator/console-operator-5ccd6b445f-gvb4l" Aug 04 15:57:25.153280 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:25.153129 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 15:57:25.153280 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.153127 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5zdb\" (UniqueName: \"kubernetes.io/projected/c16f82ab-8340-49ee-b41e-5685d4592fd2-kube-api-access-t5zdb\") pod \"console-operator-5ccd6b445f-gvb4l\" (UID: \"c16f82ab-8340-49ee-b41e-5685d4592fd2\") " pod="openshift-console-operator/console-operator-5ccd6b445f-gvb4l" Aug 04 15:57:25.153280 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:25.153176 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/820a26ee-edcd-4e18-9152-69027236d7a3-metrics-certs podName:820a26ee-edcd-4e18-9152-69027236d7a3 nodeName:}" failed. No retries permitted until 2026-08-04 15:57:25.653161292 +0000 UTC m=+45.070703397 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/820a26ee-edcd-4e18-9152-69027236d7a3-metrics-certs") pod "router-default-7c65459b9d-v7hwq" (UID: "820a26ee-edcd-4e18-9152-69027236d7a3") : secret "router-metrics-certs-default" not found Aug 04 15:57:25.153280 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.153252 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97cpr\" (UniqueName: \"kubernetes.io/projected/820a26ee-edcd-4e18-9152-69027236d7a3-kube-api-access-97cpr\") pod \"router-default-7c65459b9d-v7hwq\" (UID: \"820a26ee-edcd-4e18-9152-69027236d7a3\") " pod="openshift-ingress/router-default-7c65459b9d-v7hwq" Aug 04 15:57:25.157004 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.156983 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/820a26ee-edcd-4e18-9152-69027236d7a3-default-certificate\") pod \"router-default-7c65459b9d-v7hwq\" (UID: \"820a26ee-edcd-4e18-9152-69027236d7a3\") " pod="openshift-ingress/router-default-7c65459b9d-v7hwq" Aug 04 15:57:25.157091 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.156989 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/820a26ee-edcd-4e18-9152-69027236d7a3-stats-auth\") pod \"router-default-7c65459b9d-v7hwq\" (UID: \"820a26ee-edcd-4e18-9152-69027236d7a3\") " pod="openshift-ingress/router-default-7c65459b9d-v7hwq" Aug 04 15:57:25.164012 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.163988 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzxlz\" (UniqueName: \"kubernetes.io/projected/f715e6cb-4601-489a-9e05-af3fbd46d394-kube-api-access-dzxlz\") pod \"network-check-source-65bdbbff45-gkbnm\" (UID: \"f715e6cb-4601-489a-9e05-af3fbd46d394\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-gkbnm" Aug 04 15:57:25.164423 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.164404 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97cpr\" (UniqueName: \"kubernetes.io/projected/820a26ee-edcd-4e18-9152-69027236d7a3-kube-api-access-97cpr\") pod \"router-default-7c65459b9d-v7hwq\" (UID: \"820a26ee-edcd-4e18-9152-69027236d7a3\") " pod="openshift-ingress/router-default-7c65459b9d-v7hwq" Aug 04 15:57:25.223760 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.223719 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-gkbnm" Aug 04 15:57:25.254012 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.253987 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5zdb\" (UniqueName: \"kubernetes.io/projected/c16f82ab-8340-49ee-b41e-5685d4592fd2-kube-api-access-t5zdb\") pod \"console-operator-5ccd6b445f-gvb4l\" (UID: \"c16f82ab-8340-49ee-b41e-5685d4592fd2\") " pod="openshift-console-operator/console-operator-5ccd6b445f-gvb4l" Aug 04 15:57:25.254146 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.254071 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c16f82ab-8340-49ee-b41e-5685d4592fd2-serving-cert\") pod \"console-operator-5ccd6b445f-gvb4l\" (UID: \"c16f82ab-8340-49ee-b41e-5685d4592fd2\") " pod="openshift-console-operator/console-operator-5ccd6b445f-gvb4l" Aug 04 15:57:25.254146 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.254102 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvpfm\" (UniqueName: \"kubernetes.io/projected/c06a604b-4484-4ead-a88b-d64926af2a2b-kube-api-access-qvpfm\") pod \"volume-data-source-validator-5b8776669-d97mf\" (UID: \"c06a604b-4484-4ead-a88b-d64926af2a2b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-d97mf" Aug 04 15:57:25.254261 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.254154 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c16f82ab-8340-49ee-b41e-5685d4592fd2-config\") pod \"console-operator-5ccd6b445f-gvb4l\" (UID: \"c16f82ab-8340-49ee-b41e-5685d4592fd2\") " pod="openshift-console-operator/console-operator-5ccd6b445f-gvb4l" Aug 04 15:57:25.254261 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.254207 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c16f82ab-8340-49ee-b41e-5685d4592fd2-trusted-ca\") pod \"console-operator-5ccd6b445f-gvb4l\" (UID: \"c16f82ab-8340-49ee-b41e-5685d4592fd2\") " pod="openshift-console-operator/console-operator-5ccd6b445f-gvb4l" Aug 04 15:57:25.255057 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.255031 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c16f82ab-8340-49ee-b41e-5685d4592fd2-config\") pod \"console-operator-5ccd6b445f-gvb4l\" (UID: \"c16f82ab-8340-49ee-b41e-5685d4592fd2\") " pod="openshift-console-operator/console-operator-5ccd6b445f-gvb4l" Aug 04 15:57:25.255308 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.255248 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c16f82ab-8340-49ee-b41e-5685d4592fd2-trusted-ca\") pod \"console-operator-5ccd6b445f-gvb4l\" (UID: \"c16f82ab-8340-49ee-b41e-5685d4592fd2\") " pod="openshift-console-operator/console-operator-5ccd6b445f-gvb4l" Aug 04 15:57:25.256590 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.256569 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c16f82ab-8340-49ee-b41e-5685d4592fd2-serving-cert\") pod \"console-operator-5ccd6b445f-gvb4l\" (UID: \"c16f82ab-8340-49ee-b41e-5685d4592fd2\") " pod="openshift-console-operator/console-operator-5ccd6b445f-gvb4l" Aug 04 15:57:25.262944 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.262925 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvpfm\" (UniqueName: \"kubernetes.io/projected/c06a604b-4484-4ead-a88b-d64926af2a2b-kube-api-access-qvpfm\") pod \"volume-data-source-validator-5b8776669-d97mf\" (UID: \"c06a604b-4484-4ead-a88b-d64926af2a2b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-d97mf" Aug 04 15:57:25.263033 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.262928 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5zdb\" (UniqueName: \"kubernetes.io/projected/c16f82ab-8340-49ee-b41e-5685d4592fd2-kube-api-access-t5zdb\") pod \"console-operator-5ccd6b445f-gvb4l\" (UID: \"c16f82ab-8340-49ee-b41e-5685d4592fd2\") " pod="openshift-console-operator/console-operator-5ccd6b445f-gvb4l" Aug 04 15:57:25.304012 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.303976 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-d97mf" Aug 04 15:57:25.327903 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.327870 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-gvb4l" Aug 04 15:57:25.368977 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.368950 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-gkbnm"] Aug 04 15:57:25.373033 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:57:25.372999 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf715e6cb_4601_489a_9e05_af3fbd46d394.slice/crio-11661c8be95d4b6935fd7988537b9d1528285965bf43841cb0b0e722287d0614 WatchSource:0}: Error finding container 11661c8be95d4b6935fd7988537b9d1528285965bf43841cb0b0e722287d0614: Status 404 returned error can't find the container with id 11661c8be95d4b6935fd7988537b9d1528285965bf43841cb0b0e722287d0614 Aug 04 15:57:25.464669 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.464637 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-d97mf"] Aug 04 15:57:25.468425 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:57:25.468395 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc06a604b_4484_4ead_a88b_d64926af2a2b.slice/crio-b8564205f2d201184d89dd99b0a5e5b63d886b634b6c22b9e722294408ad205f WatchSource:0}: Error finding container b8564205f2d201184d89dd99b0a5e5b63d886b634b6c22b9e722294408ad205f: Status 404 returned error can't find the container with id b8564205f2d201184d89dd99b0a5e5b63d886b634b6c22b9e722294408ad205f Aug 04 15:57:25.493434 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.493405 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-gvb4l"] Aug 04 15:57:25.496411 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:57:25.496386 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc16f82ab_8340_49ee_b41e_5685d4592fd2.slice/crio-7c23f9b88e32190e5a45d8452b3ba957f69629f3905f8f60116e790b3a30b31a WatchSource:0}: Error finding container 7c23f9b88e32190e5a45d8452b3ba957f69629f3905f8f60116e790b3a30b31a: Status 404 returned error can't find the container with id 7c23f9b88e32190e5a45d8452b3ba957f69629f3905f8f60116e790b3a30b31a Aug 04 15:57:25.658847 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.658808 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/820a26ee-edcd-4e18-9152-69027236d7a3-service-ca-bundle\") pod \"router-default-7c65459b9d-v7hwq\" (UID: \"820a26ee-edcd-4e18-9152-69027236d7a3\") " pod="openshift-ingress/router-default-7c65459b9d-v7hwq" Aug 04 15:57:25.658995 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:25.658861 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/820a26ee-edcd-4e18-9152-69027236d7a3-metrics-certs\") pod \"router-default-7c65459b9d-v7hwq\" (UID: \"820a26ee-edcd-4e18-9152-69027236d7a3\") " pod="openshift-ingress/router-default-7c65459b9d-v7hwq" Aug 04 15:57:25.658995 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:25.658984 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 15:57:25.659086 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:25.659017 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/820a26ee-edcd-4e18-9152-69027236d7a3-service-ca-bundle podName:820a26ee-edcd-4e18-9152-69027236d7a3 nodeName:}" failed. No retries permitted until 2026-08-04 15:57:26.658997796 +0000 UTC m=+46.076539900 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/820a26ee-edcd-4e18-9152-69027236d7a3-service-ca-bundle") pod "router-default-7c65459b9d-v7hwq" (UID: "820a26ee-edcd-4e18-9152-69027236d7a3") : configmap references non-existent config key: service-ca.crt Aug 04 15:57:25.659086 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:25.659040 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/820a26ee-edcd-4e18-9152-69027236d7a3-metrics-certs podName:820a26ee-edcd-4e18-9152-69027236d7a3 nodeName:}" failed. No retries permitted until 2026-08-04 15:57:26.659034288 +0000 UTC m=+46.076576392 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/820a26ee-edcd-4e18-9152-69027236d7a3-metrics-certs") pod "router-default-7c65459b9d-v7hwq" (UID: "820a26ee-edcd-4e18-9152-69027236d7a3") : secret "router-metrics-certs-default" not found Aug 04 15:57:26.321976 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:26.321941 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-d97mf" event={"ID":"c06a604b-4484-4ead-a88b-d64926af2a2b","Type":"ContainerStarted","Data":"b8564205f2d201184d89dd99b0a5e5b63d886b634b6c22b9e722294408ad205f"} Aug 04 15:57:26.323295 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:26.323239 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-gkbnm" event={"ID":"f715e6cb-4601-489a-9e05-af3fbd46d394","Type":"ContainerStarted","Data":"11661c8be95d4b6935fd7988537b9d1528285965bf43841cb0b0e722287d0614"} Aug 04 15:57:26.324521 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:26.324471 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-gvb4l" event={"ID":"c16f82ab-8340-49ee-b41e-5685d4592fd2","Type":"ContainerStarted","Data":"7c23f9b88e32190e5a45d8452b3ba957f69629f3905f8f60116e790b3a30b31a"} Aug 04 15:57:26.666766 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:26.666684 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/820a26ee-edcd-4e18-9152-69027236d7a3-service-ca-bundle\") pod \"router-default-7c65459b9d-v7hwq\" (UID: \"820a26ee-edcd-4e18-9152-69027236d7a3\") " pod="openshift-ingress/router-default-7c65459b9d-v7hwq" Aug 04 15:57:26.666766 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:26.666734 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/820a26ee-edcd-4e18-9152-69027236d7a3-metrics-certs\") pod \"router-default-7c65459b9d-v7hwq\" (UID: \"820a26ee-edcd-4e18-9152-69027236d7a3\") " pod="openshift-ingress/router-default-7c65459b9d-v7hwq" Aug 04 15:57:26.666948 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:26.666873 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/820a26ee-edcd-4e18-9152-69027236d7a3-service-ca-bundle podName:820a26ee-edcd-4e18-9152-69027236d7a3 nodeName:}" failed. No retries permitted until 2026-08-04 15:57:28.666855881 +0000 UTC m=+48.084398006 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/820a26ee-edcd-4e18-9152-69027236d7a3-service-ca-bundle") pod "router-default-7c65459b9d-v7hwq" (UID: "820a26ee-edcd-4e18-9152-69027236d7a3") : configmap references non-existent config key: service-ca.crt Aug 04 15:57:26.666948 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:26.666878 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 15:57:26.666948 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:26.666917 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/820a26ee-edcd-4e18-9152-69027236d7a3-metrics-certs podName:820a26ee-edcd-4e18-9152-69027236d7a3 nodeName:}" failed. No retries permitted until 2026-08-04 15:57:28.666910439 +0000 UTC m=+48.084452547 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/820a26ee-edcd-4e18-9152-69027236d7a3-metrics-certs") pod "router-default-7c65459b9d-v7hwq" (UID: "820a26ee-edcd-4e18-9152-69027236d7a3") : secret "router-metrics-certs-default" not found Aug 04 15:57:28.330905 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:28.330864 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-d97mf" event={"ID":"c06a604b-4484-4ead-a88b-d64926af2a2b","Type":"ContainerStarted","Data":"04aa67ed75192ef03621d59d6a5629d979e38bea9c2e0e6fb52a0b66b2df58bc"} Aug 04 15:57:28.332776 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:28.332751 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-gvb4l_c16f82ab-8340-49ee-b41e-5685d4592fd2/console-operator/0.log" Aug 04 15:57:28.332908 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:28.332794 2573 generic.go:356] "Generic (PLEG): container finished" podID="c16f82ab-8340-49ee-b41e-5685d4592fd2" containerID="c5215625a4fb57a42bccb67457b00b7e3eabf31261a58204002c24207f9695f2" exitCode=255 Aug 04 15:57:28.332908 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:28.332829 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-gvb4l" event={"ID":"c16f82ab-8340-49ee-b41e-5685d4592fd2","Type":"ContainerDied","Data":"c5215625a4fb57a42bccb67457b00b7e3eabf31261a58204002c24207f9695f2"} Aug 04 15:57:28.333090 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:28.333074 2573 scope.go:117] "RemoveContainer" containerID="c5215625a4fb57a42bccb67457b00b7e3eabf31261a58204002c24207f9695f2" Aug 04 15:57:28.347601 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:28.347539 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-d97mf" podStartSLOduration=2.334935844 podStartE2EDuration="4.347522892s" totalImagesPullingTime="2.012587048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:57:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:57:28.34609871 +0000 UTC m=+47.763640875" watchObservedRunningTime="2026-08-04 15:57:28.347522892 +0000 UTC m=+47.765064996" Aug 04 15:57:28.684429 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:28.684345 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/820a26ee-edcd-4e18-9152-69027236d7a3-service-ca-bundle\") pod \"router-default-7c65459b9d-v7hwq\" (UID: \"820a26ee-edcd-4e18-9152-69027236d7a3\") " pod="openshift-ingress/router-default-7c65459b9d-v7hwq" Aug 04 15:57:28.684429 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:28.684395 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/820a26ee-edcd-4e18-9152-69027236d7a3-metrics-certs\") pod \"router-default-7c65459b9d-v7hwq\" (UID: \"820a26ee-edcd-4e18-9152-69027236d7a3\") " pod="openshift-ingress/router-default-7c65459b9d-v7hwq" Aug 04 15:57:28.684707 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:28.684527 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 15:57:28.684707 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:28.684546 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/820a26ee-edcd-4e18-9152-69027236d7a3-service-ca-bundle podName:820a26ee-edcd-4e18-9152-69027236d7a3 nodeName:}" failed. No retries permitted until 2026-08-04 15:57:32.684523483 +0000 UTC m=+52.102065593 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/820a26ee-edcd-4e18-9152-69027236d7a3-service-ca-bundle") pod "router-default-7c65459b9d-v7hwq" (UID: "820a26ee-edcd-4e18-9152-69027236d7a3") : configmap references non-existent config key: service-ca.crt Aug 04 15:57:28.684707 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:28.684594 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/820a26ee-edcd-4e18-9152-69027236d7a3-metrics-certs podName:820a26ee-edcd-4e18-9152-69027236d7a3 nodeName:}" failed. No retries permitted until 2026-08-04 15:57:32.684582351 +0000 UTC m=+52.102124462 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/820a26ee-edcd-4e18-9152-69027236d7a3-metrics-certs") pod "router-default-7c65459b9d-v7hwq" (UID: "820a26ee-edcd-4e18-9152-69027236d7a3") : secret "router-metrics-certs-default" not found Aug 04 15:57:29.337255 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:29.337232 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-gvb4l_c16f82ab-8340-49ee-b41e-5685d4592fd2/console-operator/1.log" Aug 04 15:57:29.337718 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:29.337695 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-gvb4l_c16f82ab-8340-49ee-b41e-5685d4592fd2/console-operator/0.log" Aug 04 15:57:29.337839 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:29.337734 2573 generic.go:356] "Generic (PLEG): container finished" podID="c16f82ab-8340-49ee-b41e-5685d4592fd2" containerID="bcf8fa9d2304603da643c53ef1f95e4ea42b8742f7f18fa1319335dd473fc1e0" exitCode=255 Aug 04 15:57:29.337839 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:29.337789 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-gvb4l" event={"ID":"c16f82ab-8340-49ee-b41e-5685d4592fd2","Type":"ContainerDied","Data":"bcf8fa9d2304603da643c53ef1f95e4ea42b8742f7f18fa1319335dd473fc1e0"} Aug 04 15:57:29.337964 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:29.337868 2573 scope.go:117] "RemoveContainer" containerID="c5215625a4fb57a42bccb67457b00b7e3eabf31261a58204002c24207f9695f2" Aug 04 15:57:29.338112 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:29.338094 2573 scope.go:117] "RemoveContainer" containerID="bcf8fa9d2304603da643c53ef1f95e4ea42b8742f7f18fa1319335dd473fc1e0" Aug 04 15:57:29.338309 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:29.338286 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-gvb4l_openshift-console-operator(c16f82ab-8340-49ee-b41e-5685d4592fd2)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-gvb4l" podUID="c16f82ab-8340-49ee-b41e-5685d4592fd2" Aug 04 15:57:30.342922 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:30.342886 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-gvb4l_c16f82ab-8340-49ee-b41e-5685d4592fd2/console-operator/1.log" Aug 04 15:57:30.343382 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:30.343300 2573 scope.go:117] "RemoveContainer" containerID="bcf8fa9d2304603da643c53ef1f95e4ea42b8742f7f18fa1319335dd473fc1e0" Aug 04 15:57:30.343485 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:30.343461 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-gvb4l_openshift-console-operator(c16f82ab-8340-49ee-b41e-5685d4592fd2)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-gvb4l" podUID="c16f82ab-8340-49ee-b41e-5685d4592fd2" Aug 04 15:57:30.702908 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:30.702878 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68c83082-033c-4f0a-83ad-6291422eac03-metrics-tls\") pod \"dns-default-jpd9h\" (UID: \"68c83082-033c-4f0a-83ad-6291422eac03\") " pod="openshift-dns/dns-default-jpd9h" Aug 04 15:57:30.703089 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:30.703059 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 15:57:30.703158 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:30.703145 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68c83082-033c-4f0a-83ad-6291422eac03-metrics-tls podName:68c83082-033c-4f0a-83ad-6291422eac03 nodeName:}" failed. No retries permitted until 2026-08-04 15:57:46.703113752 +0000 UTC m=+66.120655870 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68c83082-033c-4f0a-83ad-6291422eac03-metrics-tls") pod "dns-default-jpd9h" (UID: "68c83082-033c-4f0a-83ad-6291422eac03") : secret "dns-default-metrics-tls" not found Aug 04 15:57:31.195726 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:31.195638 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-zppl2"] Aug 04 15:57:31.197638 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:31.197617 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-zppl2" Aug 04 15:57:31.200649 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:31.200628 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Aug 04 15:57:31.201229 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:31.201214 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-n86q7\"" Aug 04 15:57:31.204176 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:31.204152 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Aug 04 15:57:31.215041 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:31.215003 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-zppl2"] Aug 04 15:57:31.308530 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:31.308489 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlt7h\" (UniqueName: \"kubernetes.io/projected/084fc17a-af5c-4dc9-9564-ed01e2588d4b-kube-api-access-dlt7h\") pod \"migrator-68bc745d87-zppl2\" (UID: \"084fc17a-af5c-4dc9-9564-ed01e2588d4b\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-zppl2" Aug 04 15:57:31.347245 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:31.347209 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-gkbnm" event={"ID":"f715e6cb-4601-489a-9e05-af3fbd46d394","Type":"ContainerStarted","Data":"3fc67493d55abbf8433392be0990b8898ed44755a17ab5fc9d752b9cfedee0e9"} Aug 04 15:57:31.364022 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:31.363973 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-gkbnm" podStartSLOduration=2.251426902 podStartE2EDuration="7.363954832s" totalImagesPullingTime="5.11252793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:57:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:57:31.363607972 +0000 UTC m=+50.781150097" watchObservedRunningTime="2026-08-04 15:57:31.363954832 +0000 UTC m=+50.781496959" Aug 04 15:57:31.409112 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:31.409075 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlt7h\" (UniqueName: \"kubernetes.io/projected/084fc17a-af5c-4dc9-9564-ed01e2588d4b-kube-api-access-dlt7h\") pod \"migrator-68bc745d87-zppl2\" (UID: \"084fc17a-af5c-4dc9-9564-ed01e2588d4b\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-zppl2" Aug 04 15:57:31.420472 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:31.420437 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlt7h\" (UniqueName: \"kubernetes.io/projected/084fc17a-af5c-4dc9-9564-ed01e2588d4b-kube-api-access-dlt7h\") pod \"migrator-68bc745d87-zppl2\" (UID: \"084fc17a-af5c-4dc9-9564-ed01e2588d4b\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-zppl2" Aug 04 15:57:31.506294 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:31.506258 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-zppl2" Aug 04 15:57:31.633633 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:31.633607 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-zppl2"] Aug 04 15:57:31.637137 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:57:31.637102 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod084fc17a_af5c_4dc9_9564_ed01e2588d4b.slice/crio-64727354c32364ed45bb61a25dc82372af8ba4a4c30da04548336ea20bb2457e WatchSource:0}: Error finding container 64727354c32364ed45bb61a25dc82372af8ba4a4c30da04548336ea20bb2457e: Status 404 returned error can't find the container with id 64727354c32364ed45bb61a25dc82372af8ba4a4c30da04548336ea20bb2457e Aug 04 15:57:32.354260 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:32.354213 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-zppl2" event={"ID":"084fc17a-af5c-4dc9-9564-ed01e2588d4b","Type":"ContainerStarted","Data":"64727354c32364ed45bb61a25dc82372af8ba4a4c30da04548336ea20bb2457e"} Aug 04 15:57:32.718757 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:32.718714 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/820a26ee-edcd-4e18-9152-69027236d7a3-service-ca-bundle\") pod \"router-default-7c65459b9d-v7hwq\" (UID: \"820a26ee-edcd-4e18-9152-69027236d7a3\") " pod="openshift-ingress/router-default-7c65459b9d-v7hwq" Aug 04 15:57:32.718921 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:32.718777 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/820a26ee-edcd-4e18-9152-69027236d7a3-metrics-certs\") pod \"router-default-7c65459b9d-v7hwq\" (UID: \"820a26ee-edcd-4e18-9152-69027236d7a3\") " pod="openshift-ingress/router-default-7c65459b9d-v7hwq" Aug 04 15:57:32.718968 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:32.718911 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/820a26ee-edcd-4e18-9152-69027236d7a3-service-ca-bundle podName:820a26ee-edcd-4e18-9152-69027236d7a3 nodeName:}" failed. No retries permitted until 2026-08-04 15:57:40.718889552 +0000 UTC m=+60.136431679 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/820a26ee-edcd-4e18-9152-69027236d7a3-service-ca-bundle") pod "router-default-7c65459b9d-v7hwq" (UID: "820a26ee-edcd-4e18-9152-69027236d7a3") : configmap references non-existent config key: service-ca.crt Aug 04 15:57:32.719021 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:32.718974 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 15:57:32.719055 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:32.719031 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/820a26ee-edcd-4e18-9152-69027236d7a3-metrics-certs podName:820a26ee-edcd-4e18-9152-69027236d7a3 nodeName:}" failed. No retries permitted until 2026-08-04 15:57:40.719016621 +0000 UTC m=+60.136558725 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/820a26ee-edcd-4e18-9152-69027236d7a3-metrics-certs") pod "router-default-7c65459b9d-v7hwq" (UID: "820a26ee-edcd-4e18-9152-69027236d7a3") : secret "router-metrics-certs-default" not found Aug 04 15:57:33.359163 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:33.359124 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-zppl2" event={"ID":"084fc17a-af5c-4dc9-9564-ed01e2588d4b","Type":"ContainerStarted","Data":"a184ef698951f85cd797fabb363c5299eaeb84caa712d680b87e3e1565e2ec9b"} Aug 04 15:57:33.359579 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:33.359174 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-zppl2" event={"ID":"084fc17a-af5c-4dc9-9564-ed01e2588d4b","Type":"ContainerStarted","Data":"de4a119a831fcc082d358de3c13a1b619ec01df18804797a853489ae3dfe4433"} Aug 04 15:57:33.381494 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:33.381438 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-zppl2" podStartSLOduration=1.294561196 podStartE2EDuration="2.381421949s" totalImagesPullingTime="1.086860753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:57:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:57:33.380130621 +0000 UTC m=+52.797672748" watchObservedRunningTime="2026-08-04 15:57:33.381421949 +0000 UTC m=+52.798964075" Aug 04 15:57:34.953481 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:34.953453 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zfqf9_482db4e4-6c5f-4c1e-9666-88f470fe67db/dns-node-resolver/0.log" Aug 04 15:57:35.328338 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:35.328236 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-gvb4l" Aug 04 15:57:35.328338 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:35.328310 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-gvb4l" Aug 04 15:57:35.328738 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:35.328722 2573 scope.go:117] "RemoveContainer" containerID="bcf8fa9d2304603da643c53ef1f95e4ea42b8742f7f18fa1319335dd473fc1e0" Aug 04 15:57:35.328907 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:35.328886 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-gvb4l_openshift-console-operator(c16f82ab-8340-49ee-b41e-5685d4592fd2)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-gvb4l" podUID="c16f82ab-8340-49ee-b41e-5685d4592fd2" Aug 04 15:57:35.950975 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:35.950943 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rz25w_c10e0ff5-2215-4032-976a-907e6dccb2d3/node-ca/0.log" Aug 04 15:57:36.550495 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:36.550474 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-q6m46_80648f1b-68e3-44d7-b0f9-8e93d70c9413/serve-healthcheck-canary/0.log" Aug 04 15:57:36.952074 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:36.952048 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-zppl2_084fc17a-af5c-4dc9-9564-ed01e2588d4b/migrator/0.log" Aug 04 15:57:37.150531 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:37.150497 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-zppl2_084fc17a-af5c-4dc9-9564-ed01e2588d4b/graceful-termination/0.log" Aug 04 15:57:40.782379 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:40.782344 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/820a26ee-edcd-4e18-9152-69027236d7a3-service-ca-bundle\") pod \"router-default-7c65459b9d-v7hwq\" (UID: \"820a26ee-edcd-4e18-9152-69027236d7a3\") " pod="openshift-ingress/router-default-7c65459b9d-v7hwq" Aug 04 15:57:40.782785 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:40.782392 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/820a26ee-edcd-4e18-9152-69027236d7a3-metrics-certs\") pod \"router-default-7c65459b9d-v7hwq\" (UID: \"820a26ee-edcd-4e18-9152-69027236d7a3\") " pod="openshift-ingress/router-default-7c65459b9d-v7hwq" Aug 04 15:57:40.782994 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:40.782975 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/820a26ee-edcd-4e18-9152-69027236d7a3-service-ca-bundle\") pod \"router-default-7c65459b9d-v7hwq\" (UID: \"820a26ee-edcd-4e18-9152-69027236d7a3\") " pod="openshift-ingress/router-default-7c65459b9d-v7hwq" Aug 04 15:57:40.784793 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:40.784775 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/820a26ee-edcd-4e18-9152-69027236d7a3-metrics-certs\") pod \"router-default-7c65459b9d-v7hwq\" (UID: \"820a26ee-edcd-4e18-9152-69027236d7a3\") " pod="openshift-ingress/router-default-7c65459b9d-v7hwq" Aug 04 15:57:40.839418 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:40.839383 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7c65459b9d-v7hwq" Aug 04 15:57:41.004440 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:41.004406 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7c65459b9d-v7hwq"] Aug 04 15:57:41.009882 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:57:41.009844 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod820a26ee_edcd_4e18_9152_69027236d7a3.slice/crio-f95609568f0a9a4b8d75ac19c7538f3a421c62d568704bbe7f50be72253f6e49 WatchSource:0}: Error finding container f95609568f0a9a4b8d75ac19c7538f3a421c62d568704bbe7f50be72253f6e49: Status 404 returned error can't find the container with id f95609568f0a9a4b8d75ac19c7538f3a421c62d568704bbe7f50be72253f6e49 Aug 04 15:57:41.274047 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:41.274020 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mtbc5" Aug 04 15:57:41.386368 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:41.386273 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7c65459b9d-v7hwq" event={"ID":"820a26ee-edcd-4e18-9152-69027236d7a3","Type":"ContainerStarted","Data":"fa1bb96dbd268db95e4df9d57cd42aba5df2fac122e9f5aabd6817d1564ca4a9"} Aug 04 15:57:41.386368 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:41.386331 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7c65459b9d-v7hwq" event={"ID":"820a26ee-edcd-4e18-9152-69027236d7a3","Type":"ContainerStarted","Data":"f95609568f0a9a4b8d75ac19c7538f3a421c62d568704bbe7f50be72253f6e49"} Aug 04 15:57:41.413669 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:41.413619 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7c65459b9d-v7hwq" podStartSLOduration=17.41360274 podStartE2EDuration="17.41360274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:57:41.406586742 +0000 UTC m=+60.824128861" watchObservedRunningTime="2026-08-04 15:57:41.41360274 +0000 UTC m=+60.831144865" Aug 04 15:57:41.840048 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:41.840009 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7c65459b9d-v7hwq" Aug 04 15:57:41.842620 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:41.842596 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7c65459b9d-v7hwq" Aug 04 15:57:42.389645 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:42.389620 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7c65459b9d-v7hwq" Aug 04 15:57:42.390894 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:42.390876 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7c65459b9d-v7hwq" Aug 04 15:57:46.073075 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:46.073046 2573 scope.go:117] "RemoveContainer" containerID="bcf8fa9d2304603da643c53ef1f95e4ea42b8742f7f18fa1319335dd473fc1e0" Aug 04 15:57:46.403597 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:46.403515 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-gvb4l_c16f82ab-8340-49ee-b41e-5685d4592fd2/console-operator/2.log" Aug 04 15:57:46.403959 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:46.403945 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-gvb4l_c16f82ab-8340-49ee-b41e-5685d4592fd2/console-operator/1.log" Aug 04 15:57:46.404009 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:46.403976 2573 generic.go:356] "Generic (PLEG): container finished" podID="c16f82ab-8340-49ee-b41e-5685d4592fd2" containerID="d3c0551b28931748342c6451a79d5994395d2504bd79d2be9e1cb3ee1f20028b" exitCode=255 Aug 04 15:57:46.404066 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:46.404048 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-gvb4l" event={"ID":"c16f82ab-8340-49ee-b41e-5685d4592fd2","Type":"ContainerDied","Data":"d3c0551b28931748342c6451a79d5994395d2504bd79d2be9e1cb3ee1f20028b"} Aug 04 15:57:46.404112 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:46.404105 2573 scope.go:117] "RemoveContainer" containerID="bcf8fa9d2304603da643c53ef1f95e4ea42b8742f7f18fa1319335dd473fc1e0" Aug 04 15:57:46.404385 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:46.404369 2573 scope.go:117] "RemoveContainer" containerID="d3c0551b28931748342c6451a79d5994395d2504bd79d2be9e1cb3ee1f20028b" Aug 04 15:57:46.404579 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:46.404540 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-5ccd6b445f-gvb4l_openshift-console-operator(c16f82ab-8340-49ee-b41e-5685d4592fd2)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-gvb4l" podUID="c16f82ab-8340-49ee-b41e-5685d4592fd2" Aug 04 15:57:46.728641 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:46.728599 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68c83082-033c-4f0a-83ad-6291422eac03-metrics-tls\") pod \"dns-default-jpd9h\" (UID: \"68c83082-033c-4f0a-83ad-6291422eac03\") " pod="openshift-dns/dns-default-jpd9h" Aug 04 15:57:46.730793 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:46.730775 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68c83082-033c-4f0a-83ad-6291422eac03-metrics-tls\") pod \"dns-default-jpd9h\" (UID: \"68c83082-033c-4f0a-83ad-6291422eac03\") " pod="openshift-dns/dns-default-jpd9h" Aug 04 15:57:46.829085 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:46.829049 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb454294-c3a4-4d8d-b705-95350ed95c13-metrics-certs\") pod \"network-metrics-daemon-rcb4z\" (UID: \"fb454294-c3a4-4d8d-b705-95350ed95c13\") " pod="openshift-multus/network-metrics-daemon-rcb4z" Aug 04 15:57:46.829085 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:46.829088 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bmm5\" (UniqueName: \"kubernetes.io/projected/cd2d6392-5d3a-49f6-be6a-954d6aaffba6-kube-api-access-6bmm5\") pod \"network-check-target-bc6jp\" (UID: \"cd2d6392-5d3a-49f6-be6a-954d6aaffba6\") " pod="openshift-network-diagnostics/network-check-target-bc6jp" Aug 04 15:57:46.829338 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:46.829112 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e05ec517-319e-4360-8a0f-6133fb238192-original-pull-secret\") pod \"global-pull-secret-syncer-q2zlj\" (UID: \"e05ec517-319e-4360-8a0f-6133fb238192\") " pod="kube-system/global-pull-secret-syncer-q2zlj" Aug 04 15:57:46.831501 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:46.831481 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 15:57:46.831604 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:46.831481 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 15:57:46.831767 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:46.831744 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bmm5\" (UniqueName: \"kubernetes.io/projected/cd2d6392-5d3a-49f6-be6a-954d6aaffba6-kube-api-access-6bmm5\") pod \"network-check-target-bc6jp\" (UID: \"cd2d6392-5d3a-49f6-be6a-954d6aaffba6\") " pod="openshift-network-diagnostics/network-check-target-bc6jp" Aug 04 15:57:46.841352 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:46.841322 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb454294-c3a4-4d8d-b705-95350ed95c13-metrics-certs\") pod \"network-metrics-daemon-rcb4z\" (UID: \"fb454294-c3a4-4d8d-b705-95350ed95c13\") " pod="openshift-multus/network-metrics-daemon-rcb4z" Aug 04 15:57:46.841457 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:46.841421 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e05ec517-319e-4360-8a0f-6133fb238192-original-pull-secret\") pod \"global-pull-secret-syncer-q2zlj\" (UID: \"e05ec517-319e-4360-8a0f-6133fb238192\") " pod="kube-system/global-pull-secret-syncer-q2zlj" Aug 04 15:57:46.892676 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:46.892638 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2zlj" Aug 04 15:57:46.911197 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:46.911174 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xj8nj\"" Aug 04 15:57:46.916503 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:46.916481 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bfprq\"" Aug 04 15:57:46.919411 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:46.919391 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc6jp" Aug 04 15:57:46.924715 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:46.924658 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rcb4z" Aug 04 15:57:47.033266 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:47.033237 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fnzc4\"" Aug 04 15:57:47.039293 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:47.039262 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jpd9h" Aug 04 15:57:47.059642 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:47.059604 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q2zlj"] Aug 04 15:57:47.064104 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:57:47.064079 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode05ec517_319e_4360_8a0f_6133fb238192.slice/crio-1c299acf505d1ab38debf6f92b1f55f08a5210c8d81edef0a7aa6eba17031f9f WatchSource:0}: Error finding container 1c299acf505d1ab38debf6f92b1f55f08a5210c8d81edef0a7aa6eba17031f9f: Status 404 returned error can't find the container with id 1c299acf505d1ab38debf6f92b1f55f08a5210c8d81edef0a7aa6eba17031f9f Aug 04 15:57:47.078103 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:47.078076 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bc6jp"] Aug 04 15:57:47.082379 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:57:47.082351 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd2d6392_5d3a_49f6_be6a_954d6aaffba6.slice/crio-551f41a07c4a3227d1eac364ea911a7521425d9a604b9fb29c1810f0c4a7832d WatchSource:0}: Error finding container 551f41a07c4a3227d1eac364ea911a7521425d9a604b9fb29c1810f0c4a7832d: Status 404 returned error can't find the container with id 551f41a07c4a3227d1eac364ea911a7521425d9a604b9fb29c1810f0c4a7832d Aug 04 15:57:47.098435 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:47.098126 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rcb4z"] Aug 04 15:57:47.103434 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:57:47.103395 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb454294_c3a4_4d8d_b705_95350ed95c13.slice/crio-43d54c382f192dcd3f5669801abce4ac30c5c37feefc120293a678f8cc6f12fd WatchSource:0}: Error finding container 43d54c382f192dcd3f5669801abce4ac30c5c37feefc120293a678f8cc6f12fd: Status 404 returned error can't find the container with id 43d54c382f192dcd3f5669801abce4ac30c5c37feefc120293a678f8cc6f12fd Aug 04 15:57:47.198830 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:47.198798 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jpd9h"] Aug 04 15:57:47.203197 ip-10-0-141-210 kubenswrapper[2573]: W0804 15:57:47.203172 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68c83082_033c_4f0a_83ad_6291422eac03.slice/crio-4db11f3d4fcd08bd3472cdb1b77daf2e30cfce868f663319eb08e5db2e070f62 WatchSource:0}: Error finding container 4db11f3d4fcd08bd3472cdb1b77daf2e30cfce868f663319eb08e5db2e070f62: Status 404 returned error can't find the container with id 4db11f3d4fcd08bd3472cdb1b77daf2e30cfce868f663319eb08e5db2e070f62 Aug 04 15:57:47.408463 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:47.408374 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rcb4z" event={"ID":"fb454294-c3a4-4d8d-b705-95350ed95c13","Type":"ContainerStarted","Data":"43d54c382f192dcd3f5669801abce4ac30c5c37feefc120293a678f8cc6f12fd"} Aug 04 15:57:47.409321 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:47.409298 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q2zlj" event={"ID":"e05ec517-319e-4360-8a0f-6133fb238192","Type":"ContainerStarted","Data":"1c299acf505d1ab38debf6f92b1f55f08a5210c8d81edef0a7aa6eba17031f9f"} Aug 04 15:57:47.410664 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:47.410648 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-gvb4l_c16f82ab-8340-49ee-b41e-5685d4592fd2/console-operator/2.log" Aug 04 15:57:47.411647 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:47.411628 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jpd9h" event={"ID":"68c83082-033c-4f0a-83ad-6291422eac03","Type":"ContainerStarted","Data":"4db11f3d4fcd08bd3472cdb1b77daf2e30cfce868f663319eb08e5db2e070f62"} Aug 04 15:57:47.412784 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:47.412767 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bc6jp" event={"ID":"cd2d6392-5d3a-49f6-be6a-954d6aaffba6","Type":"ContainerStarted","Data":"41942c343d677e7f74e59345413fa012eee038efc12d792d21e092cda458f051"} Aug 04 15:57:47.412857 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:47.412800 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bc6jp" event={"ID":"cd2d6392-5d3a-49f6-be6a-954d6aaffba6","Type":"ContainerStarted","Data":"551f41a07c4a3227d1eac364ea911a7521425d9a604b9fb29c1810f0c4a7832d"} Aug 04 15:57:47.412907 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:47.412900 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bc6jp" Aug 04 15:57:47.431830 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:47.431728 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bc6jp" podStartSLOduration=67.431711007 podStartE2EDuration="1m7.431711007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:57:47.430483772 +0000 UTC m=+66.848025923" watchObservedRunningTime="2026-08-04 15:57:47.431711007 +0000 UTC m=+66.849253135" Aug 04 15:57:49.423621 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:49.423523 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rcb4z" event={"ID":"fb454294-c3a4-4d8d-b705-95350ed95c13","Type":"ContainerStarted","Data":"1b3f9367925af235f3f71766b46d6ede1de8d7b705bb26869cc007fe280afcbe"} Aug 04 15:57:49.423621 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:49.423598 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rcb4z" event={"ID":"fb454294-c3a4-4d8d-b705-95350ed95c13","Type":"ContainerStarted","Data":"1552463bea91c89964a011fb65b31bdf69ad585cf33bd1468302c8a801d86497"} Aug 04 15:57:49.441749 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:49.441686 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rcb4z" podStartSLOduration=66.749453126 podStartE2EDuration="1m8.441661884s" totalImagesPullingTime="1.692208758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:56:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:57:49.441102382 +0000 UTC m=+68.858644509" watchObservedRunningTime="2026-08-04 15:57:49.441661884 +0000 UTC m=+68.859204012" Aug 04 15:57:52.435297 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:52.435258 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q2zlj" event={"ID":"e05ec517-319e-4360-8a0f-6133fb238192","Type":"ContainerStarted","Data":"68ff6e70b74fdfc5de15c57d6c48c4d6ad742d87500acc699a8a6ee3d5ff28e0"} Aug 04 15:57:52.436937 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:52.436910 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jpd9h" event={"ID":"68c83082-033c-4f0a-83ad-6291422eac03","Type":"ContainerStarted","Data":"652671d6aa4ad032643bffd7802c75a1f11cc12e6b301b52ebe6d251a3aa7db6"} Aug 04 15:57:52.437056 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:52.436959 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jpd9h" event={"ID":"68c83082-033c-4f0a-83ad-6291422eac03","Type":"ContainerStarted","Data":"368dc4c8a7ff41696317994bcc643db94191572dc22910bccb5149af8d18cc1d"} Aug 04 15:57:52.437056 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:52.437045 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jpd9h" Aug 04 15:57:52.450632 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:52.450586 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-q2zlj" podStartSLOduration=66.975046716 podStartE2EDuration="1m11.450569401s" totalImagesPullingTime="4.475522685s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 15:56:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:57:52.449675382 +0000 UTC m=+71.867217519" watchObservedRunningTime="2026-08-04 15:57:52.450569401 +0000 UTC m=+71.868111520" Aug 04 15:57:52.466605 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:52.466533 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jpd9h" podStartSLOduration=34.139443523 podStartE2EDuration="38.466519s" totalImagesPullingTime="4.327075477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:57:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:57:52.465783993 +0000 UTC m=+71.883326120" watchObservedRunningTime="2026-08-04 15:57:52.466519 +0000 UTC m=+71.884061126" Aug 04 15:57:55.328494 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:55.328455 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-gvb4l" Aug 04 15:57:55.329031 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:55.328513 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-gvb4l" Aug 04 15:57:55.329031 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:57:55.328941 2573 scope.go:117] "RemoveContainer" containerID="d3c0551b28931748342c6451a79d5994395d2504bd79d2be9e1cb3ee1f20028b" Aug 04 15:57:55.329157 ip-10-0-141-210 kubenswrapper[2573]: E0804 15:57:55.329135 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-5ccd6b445f-gvb4l_openshift-console-operator(c16f82ab-8340-49ee-b41e-5685d4592fd2)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-gvb4l" podUID="c16f82ab-8340-49ee-b41e-5685d4592fd2" Aug 04 15:58:02.442642 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:58:02.442526 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jpd9h" Aug 04 15:58:07.073754 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:58:07.073719 2573 scope.go:117] "RemoveContainer" containerID="d3c0551b28931748342c6451a79d5994395d2504bd79d2be9e1cb3ee1f20028b" Aug 04 15:58:07.489619 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:58:07.489592 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-gvb4l_c16f82ab-8340-49ee-b41e-5685d4592fd2/console-operator/2.log" Aug 04 15:58:07.489786 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:58:07.489675 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-gvb4l" event={"ID":"c16f82ab-8340-49ee-b41e-5685d4592fd2","Type":"ContainerStarted","Data":"48740c2730fffdd00b4ab65c3f7a66864664256c94c492ce78f7389ca5672f3a"} Aug 04 15:58:07.490033 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:58:07.490011 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-gvb4l" Aug 04 15:58:07.510379 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:58:07.510328 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-5ccd6b445f-gvb4l" podStartSLOduration=41.518976778 podStartE2EDuration="43.510309809s" totalImagesPullingTime="1.991333031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:57:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:58:07.509437339 +0000 UTC m=+86.926979466" watchObservedRunningTime="2026-08-04 15:58:07.510309809 +0000 UTC m=+86.927851934" Aug 04 15:58:07.745137 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:58:07.745060 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-5ccd6b445f-gvb4l" Aug 04 15:58:18.420100 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:58:18.420067 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bc6jp" Aug 04 15:58:44.285803 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:58:44.285770 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-gvb4l_c16f82ab-8340-49ee-b41e-5685d4592fd2/console-operator/2.log" Aug 04 15:58:44.488251 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:58:44.488217 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-gvb4l_c16f82ab-8340-49ee-b41e-5685d4592fd2/console-operator/3.log" Aug 04 15:58:45.086526 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:58:45.086502 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jpd9h_68c83082-033c-4f0a-83ad-6291422eac03/dns/0.log" Aug 04 15:58:45.287334 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:58:45.287300 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jpd9h_68c83082-033c-4f0a-83ad-6291422eac03/kube-rbac-proxy/0.log" Aug 04 15:58:46.286175 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:58:46.286147 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zfqf9_482db4e4-6c5f-4c1e-9666-88f470fe67db/dns-node-resolver/0.log" Aug 04 15:58:47.286090 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:58:47.286067 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rz25w_c10e0ff5-2215-4032-976a-907e6dccb2d3/node-ca/0.log" Aug 04 15:58:47.487318 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:58:47.487287 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7c65459b9d-v7hwq_820a26ee-edcd-4e18-9152-69027236d7a3/router/0.log" Aug 04 15:58:47.886562 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:58:47.886532 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-q6m46_80648f1b-68e3-44d7-b0f9-8e93d70c9413/serve-healthcheck-canary/0.log" Aug 04 15:58:48.287047 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:58:48.287022 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-zppl2_084fc17a-af5c-4dc9-9564-ed01e2588d4b/migrator/0.log" Aug 04 15:58:48.486984 ip-10-0-141-210 kubenswrapper[2573]: I0804 15:58:48.486957 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-zppl2_084fc17a-af5c-4dc9-9564-ed01e2588d4b/graceful-termination/0.log" Aug 04 16:00:31.603606 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:31.603564 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69f87bcbb4-sqsj7"] Aug 04 16:00:31.605657 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:31.605633 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69f87bcbb4-sqsj7" Aug 04 16:00:31.608772 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:00:31.608730 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\" is forbidden: User \"system:node:ip-10-0-141-210.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-141-210.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" type="*v1.Secret" Aug 04 16:00:31.608932 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:00:31.608903 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"cluster-proxy-proxy-agent-69f87bcbb4-sqsj7\" is forbidden: User \"system:node:ip-10-0-141-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-141-210.ec2.internal' and this object" podUID="4b719d1f-474e-45c8-afba-9d6612e5dbe0" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69f87bcbb4-sqsj7" Aug 04 16:00:31.610625 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:00:31.610589 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-proxy-hub-kubeconfig\" is forbidden: User \"system:node:ip-10-0-141-210.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-141-210.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" type="*v1.Secret" Aug 04 16:00:31.610752 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:00:31.610600 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"open-cluster-management-image-pull-credentials\" is forbidden: User \"system:node:ip-10-0-141-210.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-141-210.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" type="*v1.Secret" Aug 04 16:00:31.610752 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:00:31.610649 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-141-210.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-141-210.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Aug 04 16:00:31.610752 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:00:31.610660 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-141-210.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-141-210.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Aug 04 16:00:31.610752 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:00:31.610675 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-proxy-service-proxy-server-certificates\" is forbidden: User \"system:node:ip-10-0-141-210.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-141-210.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" type="*v1.Secret" Aug 04 16:00:31.610917 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:00:31.610782 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-proxy-ca\" is forbidden: User \"system:node:ip-10-0-141-210.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-141-210.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" type="*v1.Secret" Aug 04 16:00:31.656936 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:31.656905 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69f87bcbb4-sqsj7"] Aug 04 16:00:31.789108 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:31.789074 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pbpg\" (UniqueName: \"kubernetes.io/projected/4b719d1f-474e-45c8-afba-9d6612e5dbe0-kube-api-access-8pbpg\") pod \"cluster-proxy-proxy-agent-69f87bcbb4-sqsj7\" (UID: \"4b719d1f-474e-45c8-afba-9d6612e5dbe0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69f87bcbb4-sqsj7" Aug 04 16:00:31.789300 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:31.789117 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b719d1f-474e-45c8-afba-9d6612e5dbe0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-69f87bcbb4-sqsj7\" (UID: \"4b719d1f-474e-45c8-afba-9d6612e5dbe0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69f87bcbb4-sqsj7" Aug 04 16:00:31.789300 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:31.789136 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4b719d1f-474e-45c8-afba-9d6612e5dbe0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-69f87bcbb4-sqsj7\" (UID: \"4b719d1f-474e-45c8-afba-9d6612e5dbe0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69f87bcbb4-sqsj7" Aug 04 16:00:31.789300 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:31.789186 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4b719d1f-474e-45c8-afba-9d6612e5dbe0-ca\") pod \"cluster-proxy-proxy-agent-69f87bcbb4-sqsj7\" (UID: \"4b719d1f-474e-45c8-afba-9d6612e5dbe0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69f87bcbb4-sqsj7" Aug 04 16:00:31.789300 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:31.789215 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4b719d1f-474e-45c8-afba-9d6612e5dbe0-hub\") pod \"cluster-proxy-proxy-agent-69f87bcbb4-sqsj7\" (UID: \"4b719d1f-474e-45c8-afba-9d6612e5dbe0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69f87bcbb4-sqsj7" Aug 04 16:00:31.789300 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:31.789245 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4b719d1f-474e-45c8-afba-9d6612e5dbe0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-69f87bcbb4-sqsj7\" (UID: \"4b719d1f-474e-45c8-afba-9d6612e5dbe0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69f87bcbb4-sqsj7" Aug 04 16:00:31.795094 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:31.795066 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hjc4r"] Aug 04 16:00:31.797218 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:31.797200 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hjc4r" Aug 04 16:00:31.804449 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:31.804425 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Aug 04 16:00:31.804711 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:31.804695 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Aug 04 16:00:31.805199 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:31.805185 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dngfj\"" Aug 04 16:00:31.805269 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:31.805255 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Aug 04 16:00:31.807808 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:31.807794 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Aug 04 16:00:31.828198 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:31.828170 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hjc4r"] Aug 04 16:00:31.890334 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:31.890243 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b719d1f-474e-45c8-afba-9d6612e5dbe0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-69f87bcbb4-sqsj7\" (UID: \"4b719d1f-474e-45c8-afba-9d6612e5dbe0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69f87bcbb4-sqsj7" Aug 04 16:00:31.890334 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:31.890284 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4b719d1f-474e-45c8-afba-9d6612e5dbe0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-69f87bcbb4-sqsj7\" (UID: \"4b719d1f-474e-45c8-afba-9d6612e5dbe0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69f87bcbb4-sqsj7" Aug 04 16:00:31.890334 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:31.890316 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4b719d1f-474e-45c8-afba-9d6612e5dbe0-ca\") pod \"cluster-proxy-proxy-agent-69f87bcbb4-sqsj7\" (UID: \"4b719d1f-474e-45c8-afba-9d6612e5dbe0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69f87bcbb4-sqsj7" Aug 04 16:00:31.890649 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:31.890338 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4b719d1f-474e-45c8-afba-9d6612e5dbe0-hub\") pod \"cluster-proxy-proxy-agent-69f87bcbb4-sqsj7\" (UID: \"4b719d1f-474e-45c8-afba-9d6612e5dbe0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69f87bcbb4-sqsj7" Aug 04 16:00:31.890649 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:31.890367 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4b719d1f-474e-45c8-afba-9d6612e5dbe0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-69f87bcbb4-sqsj7\" (UID: \"4b719d1f-474e-45c8-afba-9d6612e5dbe0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69f87bcbb4-sqsj7" Aug 04 16:00:31.890649 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:31.890406 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pbpg\" (UniqueName: \"kubernetes.io/projected/4b719d1f-474e-45c8-afba-9d6612e5dbe0-kube-api-access-8pbpg\") pod \"cluster-proxy-proxy-agent-69f87bcbb4-sqsj7\" (UID: \"4b719d1f-474e-45c8-afba-9d6612e5dbe0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69f87bcbb4-sqsj7" Aug 04 16:00:31.990998 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:31.990967 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e33e71a1-fc96-40e9-84e8-2e0ceda39242-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hjc4r\" (UID: \"e33e71a1-fc96-40e9-84e8-2e0ceda39242\") " pod="openshift-insights/insights-runtime-extractor-hjc4r" Aug 04 16:00:31.991178 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:31.991017 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e33e71a1-fc96-40e9-84e8-2e0ceda39242-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hjc4r\" (UID: \"e33e71a1-fc96-40e9-84e8-2e0ceda39242\") " pod="openshift-insights/insights-runtime-extractor-hjc4r" Aug 04 16:00:31.991178 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:31.991083 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e33e71a1-fc96-40e9-84e8-2e0ceda39242-data-volume\") pod \"insights-runtime-extractor-hjc4r\" (UID: \"e33e71a1-fc96-40e9-84e8-2e0ceda39242\") " pod="openshift-insights/insights-runtime-extractor-hjc4r" Aug 04 16:00:31.991178 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:31.991156 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgdgt\" (UniqueName: \"kubernetes.io/projected/e33e71a1-fc96-40e9-84e8-2e0ceda39242-kube-api-access-hgdgt\") pod \"insights-runtime-extractor-hjc4r\" (UID: \"e33e71a1-fc96-40e9-84e8-2e0ceda39242\") " pod="openshift-insights/insights-runtime-extractor-hjc4r" Aug 04 16:00:31.991306 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:31.991185 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e33e71a1-fc96-40e9-84e8-2e0ceda39242-crio-socket\") pod \"insights-runtime-extractor-hjc4r\" (UID: \"e33e71a1-fc96-40e9-84e8-2e0ceda39242\") " pod="openshift-insights/insights-runtime-extractor-hjc4r" Aug 04 16:00:32.091673 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:32.091639 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgdgt\" (UniqueName: \"kubernetes.io/projected/e33e71a1-fc96-40e9-84e8-2e0ceda39242-kube-api-access-hgdgt\") pod \"insights-runtime-extractor-hjc4r\" (UID: \"e33e71a1-fc96-40e9-84e8-2e0ceda39242\") " pod="openshift-insights/insights-runtime-extractor-hjc4r" Aug 04 16:00:32.091673 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:32.091682 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e33e71a1-fc96-40e9-84e8-2e0ceda39242-crio-socket\") pod \"insights-runtime-extractor-hjc4r\" (UID: \"e33e71a1-fc96-40e9-84e8-2e0ceda39242\") " pod="openshift-insights/insights-runtime-extractor-hjc4r" Aug 04 16:00:32.091940 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:32.091732 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e33e71a1-fc96-40e9-84e8-2e0ceda39242-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hjc4r\" (UID: \"e33e71a1-fc96-40e9-84e8-2e0ceda39242\") " pod="openshift-insights/insights-runtime-extractor-hjc4r" Aug 04 16:00:32.091940 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:32.091767 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e33e71a1-fc96-40e9-84e8-2e0ceda39242-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hjc4r\" (UID: \"e33e71a1-fc96-40e9-84e8-2e0ceda39242\") " pod="openshift-insights/insights-runtime-extractor-hjc4r" Aug 04 16:00:32.091940 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:32.091798 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e33e71a1-fc96-40e9-84e8-2e0ceda39242-data-volume\") pod \"insights-runtime-extractor-hjc4r\" (UID: \"e33e71a1-fc96-40e9-84e8-2e0ceda39242\") " pod="openshift-insights/insights-runtime-extractor-hjc4r" Aug 04 16:00:32.091940 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:32.091835 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e33e71a1-fc96-40e9-84e8-2e0ceda39242-crio-socket\") pod \"insights-runtime-extractor-hjc4r\" (UID: \"e33e71a1-fc96-40e9-84e8-2e0ceda39242\") " pod="openshift-insights/insights-runtime-extractor-hjc4r" Aug 04 16:00:32.092190 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:32.092172 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e33e71a1-fc96-40e9-84e8-2e0ceda39242-data-volume\") pod \"insights-runtime-extractor-hjc4r\" (UID: \"e33e71a1-fc96-40e9-84e8-2e0ceda39242\") " pod="openshift-insights/insights-runtime-extractor-hjc4r" Aug 04 16:00:32.092369 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:32.092348 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e33e71a1-fc96-40e9-84e8-2e0ceda39242-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hjc4r\" (UID: \"e33e71a1-fc96-40e9-84e8-2e0ceda39242\") " pod="openshift-insights/insights-runtime-extractor-hjc4r" Aug 04 16:00:32.094151 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:32.094131 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e33e71a1-fc96-40e9-84e8-2e0ceda39242-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hjc4r\" (UID: \"e33e71a1-fc96-40e9-84e8-2e0ceda39242\") " pod="openshift-insights/insights-runtime-extractor-hjc4r" Aug 04 16:00:32.118083 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:32.118051 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgdgt\" (UniqueName: \"kubernetes.io/projected/e33e71a1-fc96-40e9-84e8-2e0ceda39242-kube-api-access-hgdgt\") pod \"insights-runtime-extractor-hjc4r\" (UID: \"e33e71a1-fc96-40e9-84e8-2e0ceda39242\") " pod="openshift-insights/insights-runtime-extractor-hjc4r" Aug 04 16:00:32.406141 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:32.406080 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hjc4r" Aug 04 16:00:32.573046 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:32.572956 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hjc4r"] Aug 04 16:00:32.577346 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:00:32.577318 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode33e71a1_fc96_40e9_84e8_2e0ceda39242.slice/crio-71376b67581256a894d171671921502f2b4ee3635ded7dae1335e254ee9dc03d WatchSource:0}: Error finding container 71376b67581256a894d171671921502f2b4ee3635ded7dae1335e254ee9dc03d: Status 404 returned error can't find the container with id 71376b67581256a894d171671921502f2b4ee3635ded7dae1335e254ee9dc03d Aug 04 16:00:32.692043 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:32.688541 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Aug 04 16:00:32.743194 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:32.743160 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Aug 04 16:00:32.753110 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:32.753081 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4b719d1f-474e-45c8-afba-9d6612e5dbe0-hub\") pod \"cluster-proxy-proxy-agent-69f87bcbb4-sqsj7\" (UID: \"4b719d1f-474e-45c8-afba-9d6612e5dbe0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69f87bcbb4-sqsj7" Aug 04 16:00:32.785295 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:32.785268 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Aug 04 16:00:32.793106 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:32.793079 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4b719d1f-474e-45c8-afba-9d6612e5dbe0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-69f87bcbb4-sqsj7\" (UID: \"4b719d1f-474e-45c8-afba-9d6612e5dbe0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69f87bcbb4-sqsj7" Aug 04 16:00:32.891006 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:00:32.890971 2573 secret.go:189] Couldn't get secret open-cluster-management-agent-addon/cluster-proxy-service-proxy-server-certificates: failed to sync secret cache: timed out waiting for the condition Aug 04 16:00:32.891149 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:00:32.891051 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b719d1f-474e-45c8-afba-9d6612e5dbe0-service-proxy-server-cert podName:4b719d1f-474e-45c8-afba-9d6612e5dbe0 nodeName:}" failed. No retries permitted until 2026-08-04 16:00:33.391033912 +0000 UTC m=+232.808576016 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-proxy-server-cert" (UniqueName: "kubernetes.io/secret/4b719d1f-474e-45c8-afba-9d6612e5dbe0-service-proxy-server-cert") pod "cluster-proxy-proxy-agent-69f87bcbb4-sqsj7" (UID: "4b719d1f-474e-45c8-afba-9d6612e5dbe0") : failed to sync secret cache: timed out waiting for the condition Aug 04 16:00:32.891149 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:00:32.890980 2573 configmap.go:193] Couldn't get configMap open-cluster-management-agent-addon/openshift-service-ca.crt: failed to sync configmap cache: timed out waiting for the condition Aug 04 16:00:32.891149 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:00:32.891140 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4b719d1f-474e-45c8-afba-9d6612e5dbe0-additional-service-ca podName:4b719d1f-474e-45c8-afba-9d6612e5dbe0 nodeName:}" failed. No retries permitted until 2026-08-04 16:00:33.391127217 +0000 UTC m=+232.808669320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "additional-service-ca" (UniqueName: "kubernetes.io/configmap/4b719d1f-474e-45c8-afba-9d6612e5dbe0-additional-service-ca") pod "cluster-proxy-proxy-agent-69f87bcbb4-sqsj7" (UID: "4b719d1f-474e-45c8-afba-9d6612e5dbe0") : failed to sync configmap cache: timed out waiting for the condition Aug 04 16:00:32.891284 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:00:32.890983 2573 secret.go:189] Couldn't get secret open-cluster-management-agent-addon/cluster-proxy-ca: failed to sync secret cache: timed out waiting for the condition Aug 04 16:00:32.891284 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:00:32.891211 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b719d1f-474e-45c8-afba-9d6612e5dbe0-ca podName:4b719d1f-474e-45c8-afba-9d6612e5dbe0 nodeName:}" failed. No retries permitted until 2026-08-04 16:00:33.391196943 +0000 UTC m=+232.808739050 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ca" (UniqueName: "kubernetes.io/secret/4b719d1f-474e-45c8-afba-9d6612e5dbe0-ca") pod "cluster-proxy-proxy-agent-69f87bcbb4-sqsj7" (UID: "4b719d1f-474e-45c8-afba-9d6612e5dbe0") : failed to sync secret cache: timed out waiting for the condition Aug 04 16:00:32.930573 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:32.930516 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Aug 04 16:00:32.932123 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:32.932095 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hjc4r" event={"ID":"e33e71a1-fc96-40e9-84e8-2e0ceda39242","Type":"ContainerStarted","Data":"046894928f568576da380cdbd58a0712f7f85488c8f252f4fcd63a6638a91e5d"} Aug 04 16:00:32.932267 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:32.932184 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hjc4r" event={"ID":"e33e71a1-fc96-40e9-84e8-2e0ceda39242","Type":"ContainerStarted","Data":"71376b67581256a894d171671921502f2b4ee3635ded7dae1335e254ee9dc03d"} Aug 04 16:00:32.972562 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:32.972530 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Aug 04 16:00:33.183535 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:33.183443 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Aug 04 16:00:33.190020 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:33.190003 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pbpg\" (UniqueName: \"kubernetes.io/projected/4b719d1f-474e-45c8-afba-9d6612e5dbe0-kube-api-access-8pbpg\") pod \"cluster-proxy-proxy-agent-69f87bcbb4-sqsj7\" (UID: \"4b719d1f-474e-45c8-afba-9d6612e5dbe0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69f87bcbb4-sqsj7" Aug 04 16:00:33.197100 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:33.197081 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Aug 04 16:00:33.401659 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:33.401625 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4b719d1f-474e-45c8-afba-9d6612e5dbe0-ca\") pod \"cluster-proxy-proxy-agent-69f87bcbb4-sqsj7\" (UID: \"4b719d1f-474e-45c8-afba-9d6612e5dbe0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69f87bcbb4-sqsj7" Aug 04 16:00:33.401833 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:33.401685 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b719d1f-474e-45c8-afba-9d6612e5dbe0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-69f87bcbb4-sqsj7\" (UID: \"4b719d1f-474e-45c8-afba-9d6612e5dbe0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69f87bcbb4-sqsj7" Aug 04 16:00:33.401833 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:33.401703 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4b719d1f-474e-45c8-afba-9d6612e5dbe0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-69f87bcbb4-sqsj7\" (UID: \"4b719d1f-474e-45c8-afba-9d6612e5dbe0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69f87bcbb4-sqsj7" Aug 04 16:00:33.402301 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:33.402275 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b719d1f-474e-45c8-afba-9d6612e5dbe0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-69f87bcbb4-sqsj7\" (UID: \"4b719d1f-474e-45c8-afba-9d6612e5dbe0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69f87bcbb4-sqsj7" Aug 04 16:00:33.404051 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:33.404032 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4b719d1f-474e-45c8-afba-9d6612e5dbe0-ca\") pod \"cluster-proxy-proxy-agent-69f87bcbb4-sqsj7\" (UID: \"4b719d1f-474e-45c8-afba-9d6612e5dbe0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69f87bcbb4-sqsj7" Aug 04 16:00:33.404156 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:33.404136 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4b719d1f-474e-45c8-afba-9d6612e5dbe0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-69f87bcbb4-sqsj7\" (UID: \"4b719d1f-474e-45c8-afba-9d6612e5dbe0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69f87bcbb4-sqsj7" Aug 04 16:00:33.427243 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:33.427212 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69f87bcbb4-sqsj7" Aug 04 16:00:33.587230 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:33.587201 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69f87bcbb4-sqsj7"] Aug 04 16:00:33.587364 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:00:33.587343 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b719d1f_474e_45c8_afba_9d6612e5dbe0.slice/crio-3ac4dc6262ac7ba1428fe921da3ad4e6f0951922cb66dd04cd51410650a30aae WatchSource:0}: Error finding container 3ac4dc6262ac7ba1428fe921da3ad4e6f0951922cb66dd04cd51410650a30aae: Status 404 returned error can't find the container with id 3ac4dc6262ac7ba1428fe921da3ad4e6f0951922cb66dd04cd51410650a30aae Aug 04 16:00:33.936198 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:33.936109 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69f87bcbb4-sqsj7" event={"ID":"4b719d1f-474e-45c8-afba-9d6612e5dbe0","Type":"ContainerStarted","Data":"3ac4dc6262ac7ba1428fe921da3ad4e6f0951922cb66dd04cd51410650a30aae"} Aug 04 16:00:33.937624 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:33.937599 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hjc4r" event={"ID":"e33e71a1-fc96-40e9-84e8-2e0ceda39242","Type":"ContainerStarted","Data":"c6d4c966dab8b14d2c9e96c7268c94bd5f83e282a329712c8fc3129b83a98f40"} Aug 04 16:00:37.955677 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:37.955651 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hjc4r" event={"ID":"e33e71a1-fc96-40e9-84e8-2e0ceda39242","Type":"ContainerStarted","Data":"f76bab9d030226a7b74d3ee331dbb53aae134c5d84f8e89ffb25ffe448dc17b2"} Aug 04 16:00:37.957759 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:37.957734 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69f87bcbb4-sqsj7" event={"ID":"4b719d1f-474e-45c8-afba-9d6612e5dbe0","Type":"ContainerStarted","Data":"5ef200db9d8e1fdb8ca1347fea0a1996617e0149adbe320bd9133918d460ff59"} Aug 04 16:00:37.957875 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:37.957770 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69f87bcbb4-sqsj7" event={"ID":"4b719d1f-474e-45c8-afba-9d6612e5dbe0","Type":"ContainerStarted","Data":"82abf415d15c4d9b553e9fba092a6d0770df1f7909cf2826ffbedb676b86ef52"} Aug 04 16:00:37.957875 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:37.957781 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69f87bcbb4-sqsj7" event={"ID":"4b719d1f-474e-45c8-afba-9d6612e5dbe0","Type":"ContainerStarted","Data":"2a3de893bf32d96d5e8d082c21966be65bb61e716b53d92457fa92d805c8bf16"} Aug 04 16:00:37.976538 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:37.976491 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hjc4r" podStartSLOduration=2.143040351 podStartE2EDuration="6.97647433s" totalImagesPullingTime="4.833433979s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 16:00:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:00:37.975257287 +0000 UTC m=+237.392799414" watchObservedRunningTime="2026-08-04 16:00:37.97647433 +0000 UTC m=+237.394016456" Aug 04 16:00:37.994961 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:37.994903 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69f87bcbb4-sqsj7" podStartSLOduration=3.033264543 podStartE2EDuration="6.994888266s" totalImagesPullingTime="3.961623723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:00:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:00:37.994083561 +0000 UTC m=+237.411625688" watchObservedRunningTime="2026-08-04 16:00:37.994888266 +0000 UTC m=+237.412430391" Aug 04 16:00:39.091025 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.090992 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kjbnj"] Aug 04 16:00:39.094280 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.094264 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kjbnj" Aug 04 16:00:39.097738 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.097715 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Aug 04 16:00:39.097738 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.097731 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Aug 04 16:00:39.097892 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.097715 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Aug 04 16:00:39.097892 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.097734 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Aug 04 16:00:39.099059 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.099045 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9rss6\"" Aug 04 16:00:39.100215 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.100196 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Aug 04 16:00:39.100508 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.100272 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Aug 04 16:00:39.144016 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.143982 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3b37689f-334d-4d13-a955-bb357b709a42-root\") pod \"node-exporter-kjbnj\" (UID: \"3b37689f-334d-4d13-a955-bb357b709a42\") " pod="openshift-monitoring/node-exporter-kjbnj" Aug 04 16:00:39.144016 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.144013 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b37689f-334d-4d13-a955-bb357b709a42-sys\") pod \"node-exporter-kjbnj\" (UID: \"3b37689f-334d-4d13-a955-bb357b709a42\") " pod="openshift-monitoring/node-exporter-kjbnj" Aug 04 16:00:39.144286 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.144034 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftr8l\" (UniqueName: \"kubernetes.io/projected/3b37689f-334d-4d13-a955-bb357b709a42-kube-api-access-ftr8l\") pod \"node-exporter-kjbnj\" (UID: \"3b37689f-334d-4d13-a955-bb357b709a42\") " pod="openshift-monitoring/node-exporter-kjbnj" Aug 04 16:00:39.144286 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.144051 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b37689f-334d-4d13-a955-bb357b709a42-metrics-client-ca\") pod \"node-exporter-kjbnj\" (UID: \"3b37689f-334d-4d13-a955-bb357b709a42\") " pod="openshift-monitoring/node-exporter-kjbnj" Aug 04 16:00:39.144286 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.144143 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3b37689f-334d-4d13-a955-bb357b709a42-node-exporter-tls\") pod \"node-exporter-kjbnj\" (UID: \"3b37689f-334d-4d13-a955-bb357b709a42\") " pod="openshift-monitoring/node-exporter-kjbnj" Aug 04 16:00:39.144286 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.144171 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3b37689f-334d-4d13-a955-bb357b709a42-node-exporter-wtmp\") pod \"node-exporter-kjbnj\" (UID: \"3b37689f-334d-4d13-a955-bb357b709a42\") " pod="openshift-monitoring/node-exporter-kjbnj" Aug 04 16:00:39.144286 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.144215 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3b37689f-334d-4d13-a955-bb357b709a42-node-exporter-textfile\") pod \"node-exporter-kjbnj\" (UID: \"3b37689f-334d-4d13-a955-bb357b709a42\") " pod="openshift-monitoring/node-exporter-kjbnj" Aug 04 16:00:39.144286 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.144235 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3b37689f-334d-4d13-a955-bb357b709a42-node-exporter-accelerators-collector-config\") pod \"node-exporter-kjbnj\" (UID: \"3b37689f-334d-4d13-a955-bb357b709a42\") " pod="openshift-monitoring/node-exporter-kjbnj" Aug 04 16:00:39.144286 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.144264 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b37689f-334d-4d13-a955-bb357b709a42-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kjbnj\" (UID: \"3b37689f-334d-4d13-a955-bb357b709a42\") " pod="openshift-monitoring/node-exporter-kjbnj" Aug 04 16:00:39.245330 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.245289 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3b37689f-334d-4d13-a955-bb357b709a42-node-exporter-wtmp\") pod \"node-exporter-kjbnj\" (UID: \"3b37689f-334d-4d13-a955-bb357b709a42\") " pod="openshift-monitoring/node-exporter-kjbnj" Aug 04 16:00:39.245527 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.245353 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3b37689f-334d-4d13-a955-bb357b709a42-node-exporter-textfile\") pod \"node-exporter-kjbnj\" (UID: \"3b37689f-334d-4d13-a955-bb357b709a42\") " pod="openshift-monitoring/node-exporter-kjbnj" Aug 04 16:00:39.245527 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.245381 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3b37689f-334d-4d13-a955-bb357b709a42-node-exporter-accelerators-collector-config\") pod \"node-exporter-kjbnj\" (UID: \"3b37689f-334d-4d13-a955-bb357b709a42\") " pod="openshift-monitoring/node-exporter-kjbnj" Aug 04 16:00:39.245527 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.245416 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b37689f-334d-4d13-a955-bb357b709a42-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kjbnj\" (UID: \"3b37689f-334d-4d13-a955-bb357b709a42\") " pod="openshift-monitoring/node-exporter-kjbnj" Aug 04 16:00:39.245527 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.245451 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3b37689f-334d-4d13-a955-bb357b709a42-root\") pod \"node-exporter-kjbnj\" (UID: \"3b37689f-334d-4d13-a955-bb357b709a42\") " pod="openshift-monitoring/node-exporter-kjbnj" Aug 04 16:00:39.245527 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.245472 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b37689f-334d-4d13-a955-bb357b709a42-sys\") pod \"node-exporter-kjbnj\" (UID: \"3b37689f-334d-4d13-a955-bb357b709a42\") " pod="openshift-monitoring/node-exporter-kjbnj" Aug 04 16:00:39.245527 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.245481 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3b37689f-334d-4d13-a955-bb357b709a42-node-exporter-wtmp\") pod \"node-exporter-kjbnj\" (UID: \"3b37689f-334d-4d13-a955-bb357b709a42\") " pod="openshift-monitoring/node-exporter-kjbnj" Aug 04 16:00:39.245527 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.245503 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3b37689f-334d-4d13-a955-bb357b709a42-root\") pod \"node-exporter-kjbnj\" (UID: \"3b37689f-334d-4d13-a955-bb357b709a42\") " pod="openshift-monitoring/node-exporter-kjbnj" Aug 04 16:00:39.245527 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.245492 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftr8l\" (UniqueName: \"kubernetes.io/projected/3b37689f-334d-4d13-a955-bb357b709a42-kube-api-access-ftr8l\") pod \"node-exporter-kjbnj\" (UID: \"3b37689f-334d-4d13-a955-bb357b709a42\") " pod="openshift-monitoring/node-exporter-kjbnj" Aug 04 16:00:39.245925 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.245545 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b37689f-334d-4d13-a955-bb357b709a42-sys\") pod \"node-exporter-kjbnj\" (UID: \"3b37689f-334d-4d13-a955-bb357b709a42\") " pod="openshift-monitoring/node-exporter-kjbnj" Aug 04 16:00:39.245925 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.245575 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b37689f-334d-4d13-a955-bb357b709a42-metrics-client-ca\") pod \"node-exporter-kjbnj\" (UID: \"3b37689f-334d-4d13-a955-bb357b709a42\") " pod="openshift-monitoring/node-exporter-kjbnj" Aug 04 16:00:39.245925 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.245618 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3b37689f-334d-4d13-a955-bb357b709a42-node-exporter-tls\") pod \"node-exporter-kjbnj\" (UID: \"3b37689f-334d-4d13-a955-bb357b709a42\") " pod="openshift-monitoring/node-exporter-kjbnj" Aug 04 16:00:39.245925 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:00:39.245711 2573 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Aug 04 16:00:39.245925 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:00:39.245772 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b37689f-334d-4d13-a955-bb357b709a42-node-exporter-tls podName:3b37689f-334d-4d13-a955-bb357b709a42 nodeName:}" failed. No retries permitted until 2026-08-04 16:00:39.745753201 +0000 UTC m=+239.163295309 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/3b37689f-334d-4d13-a955-bb357b709a42-node-exporter-tls") pod "node-exporter-kjbnj" (UID: "3b37689f-334d-4d13-a955-bb357b709a42") : secret "node-exporter-tls" not found Aug 04 16:00:39.245925 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.245778 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3b37689f-334d-4d13-a955-bb357b709a42-node-exporter-textfile\") pod \"node-exporter-kjbnj\" (UID: \"3b37689f-334d-4d13-a955-bb357b709a42\") " pod="openshift-monitoring/node-exporter-kjbnj" Aug 04 16:00:39.246109 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.246047 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3b37689f-334d-4d13-a955-bb357b709a42-node-exporter-accelerators-collector-config\") pod \"node-exporter-kjbnj\" (UID: \"3b37689f-334d-4d13-a955-bb357b709a42\") " pod="openshift-monitoring/node-exporter-kjbnj" Aug 04 16:00:39.246109 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.246090 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b37689f-334d-4d13-a955-bb357b709a42-metrics-client-ca\") pod \"node-exporter-kjbnj\" (UID: \"3b37689f-334d-4d13-a955-bb357b709a42\") " pod="openshift-monitoring/node-exporter-kjbnj" Aug 04 16:00:39.247817 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.247796 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b37689f-334d-4d13-a955-bb357b709a42-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kjbnj\" (UID: \"3b37689f-334d-4d13-a955-bb357b709a42\") " pod="openshift-monitoring/node-exporter-kjbnj" Aug 04 16:00:39.262281 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.262257 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftr8l\" (UniqueName: \"kubernetes.io/projected/3b37689f-334d-4d13-a955-bb357b709a42-kube-api-access-ftr8l\") pod \"node-exporter-kjbnj\" (UID: \"3b37689f-334d-4d13-a955-bb357b709a42\") " pod="openshift-monitoring/node-exporter-kjbnj" Aug 04 16:00:39.749354 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.749317 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3b37689f-334d-4d13-a955-bb357b709a42-node-exporter-tls\") pod \"node-exporter-kjbnj\" (UID: \"3b37689f-334d-4d13-a955-bb357b709a42\") " pod="openshift-monitoring/node-exporter-kjbnj" Aug 04 16:00:39.751670 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:39.751654 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3b37689f-334d-4d13-a955-bb357b709a42-node-exporter-tls\") pod \"node-exporter-kjbnj\" (UID: \"3b37689f-334d-4d13-a955-bb357b709a42\") " pod="openshift-monitoring/node-exporter-kjbnj" Aug 04 16:00:40.003796 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:40.003714 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kjbnj" Aug 04 16:00:40.015032 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:00:40.015003 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b37689f_334d_4d13_a955_bb357b709a42.slice/crio-de7b78f44069c2685b6c902c9de6858151660add81aeddaed1ac4dfdcb88a743 WatchSource:0}: Error finding container de7b78f44069c2685b6c902c9de6858151660add81aeddaed1ac4dfdcb88a743: Status 404 returned error can't find the container with id de7b78f44069c2685b6c902c9de6858151660add81aeddaed1ac4dfdcb88a743 Aug 04 16:00:40.971028 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:40.970987 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kjbnj" event={"ID":"3b37689f-334d-4d13-a955-bb357b709a42","Type":"ContainerStarted","Data":"de7b78f44069c2685b6c902c9de6858151660add81aeddaed1ac4dfdcb88a743"} Aug 04 16:00:41.975722 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:41.975686 2573 generic.go:356] "Generic (PLEG): container finished" podID="3b37689f-334d-4d13-a955-bb357b709a42" containerID="ece5e8fb517fa9b34d22bb3851f7b02c2d7d7acd4b84c52803c750e0b8977962" exitCode=0 Aug 04 16:00:41.976096 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:41.975731 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kjbnj" event={"ID":"3b37689f-334d-4d13-a955-bb357b709a42","Type":"ContainerDied","Data":"ece5e8fb517fa9b34d22bb3851f7b02c2d7d7acd4b84c52803c750e0b8977962"} Aug 04 16:00:42.980732 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:42.980695 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kjbnj" event={"ID":"3b37689f-334d-4d13-a955-bb357b709a42","Type":"ContainerStarted","Data":"1797e61d62f76092fbece46b0cd45c0c47cd77c93b3c0b5338edb11a602a9673"} Aug 04 16:00:42.981114 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:42.980776 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kjbnj" event={"ID":"3b37689f-334d-4d13-a955-bb357b709a42","Type":"ContainerStarted","Data":"ed6a9841cf28d9d1a4f9a7c687ea135de2f4427457c58ab4ea27b29c54b8c5a9"} Aug 04 16:00:43.003346 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:43.003292 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kjbnj" podStartSLOduration=2.66153835 podStartE2EDuration="4.003277005s" totalImagesPullingTime="1.278614567s" totalInitContainerRuntime="63.124088ms" isStatefulPod=true podCreationTimestamp="2026-08-04 16:00:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:00:43.001663547 +0000 UTC m=+242.419205674" watchObservedRunningTime="2026-08-04 16:00:43.003277005 +0000 UTC m=+242.420819130" Aug 04 16:00:48.511527 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:48.511493 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54c8d7df74-vx8rr"] Aug 04 16:00:48.514416 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:48.514394 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54c8d7df74-vx8rr" Aug 04 16:00:48.516771 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:48.516747 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Aug 04 16:00:48.517227 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:48.517209 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Aug 04 16:00:48.517284 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:48.517209 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Aug 04 16:00:48.517371 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:48.517359 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Aug 04 16:00:48.517459 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:48.517445 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vmh6q\"" Aug 04 16:00:48.517624 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:48.517602 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Aug 04 16:00:48.517728 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:48.517712 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Aug 04 16:00:48.517972 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:48.517957 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Aug 04 16:00:48.522632 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:48.522613 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Aug 04 16:00:48.526192 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:48.526174 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54c8d7df74-vx8rr"] Aug 04 16:00:48.607875 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:48.607834 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjzds\" (UniqueName: \"kubernetes.io/projected/7764fad6-5190-4fab-9952-7a2d9d226b9f-kube-api-access-rjzds\") pod \"console-54c8d7df74-vx8rr\" (UID: \"7764fad6-5190-4fab-9952-7a2d9d226b9f\") " pod="openshift-console/console-54c8d7df74-vx8rr" Aug 04 16:00:48.607875 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:48.607878 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7764fad6-5190-4fab-9952-7a2d9d226b9f-console-config\") pod \"console-54c8d7df74-vx8rr\" (UID: \"7764fad6-5190-4fab-9952-7a2d9d226b9f\") " pod="openshift-console/console-54c8d7df74-vx8rr" Aug 04 16:00:48.608107 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:48.607906 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7764fad6-5190-4fab-9952-7a2d9d226b9f-oauth-serving-cert\") pod \"console-54c8d7df74-vx8rr\" (UID: \"7764fad6-5190-4fab-9952-7a2d9d226b9f\") " pod="openshift-console/console-54c8d7df74-vx8rr" Aug 04 16:00:48.608107 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:48.607974 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7764fad6-5190-4fab-9952-7a2d9d226b9f-console-serving-cert\") pod \"console-54c8d7df74-vx8rr\" (UID: \"7764fad6-5190-4fab-9952-7a2d9d226b9f\") " pod="openshift-console/console-54c8d7df74-vx8rr" Aug 04 16:00:48.608107 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:48.608006 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7764fad6-5190-4fab-9952-7a2d9d226b9f-service-ca\") pod \"console-54c8d7df74-vx8rr\" (UID: \"7764fad6-5190-4fab-9952-7a2d9d226b9f\") " pod="openshift-console/console-54c8d7df74-vx8rr" Aug 04 16:00:48.608107 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:48.608032 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7764fad6-5190-4fab-9952-7a2d9d226b9f-trusted-ca-bundle\") pod \"console-54c8d7df74-vx8rr\" (UID: \"7764fad6-5190-4fab-9952-7a2d9d226b9f\") " pod="openshift-console/console-54c8d7df74-vx8rr" Aug 04 16:00:48.608107 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:48.608047 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7764fad6-5190-4fab-9952-7a2d9d226b9f-console-oauth-config\") pod \"console-54c8d7df74-vx8rr\" (UID: \"7764fad6-5190-4fab-9952-7a2d9d226b9f\") " pod="openshift-console/console-54c8d7df74-vx8rr" Aug 04 16:00:48.708637 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:48.708601 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7764fad6-5190-4fab-9952-7a2d9d226b9f-oauth-serving-cert\") pod \"console-54c8d7df74-vx8rr\" (UID: \"7764fad6-5190-4fab-9952-7a2d9d226b9f\") " pod="openshift-console/console-54c8d7df74-vx8rr" Aug 04 16:00:48.708795 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:48.708648 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7764fad6-5190-4fab-9952-7a2d9d226b9f-console-serving-cert\") pod \"console-54c8d7df74-vx8rr\" (UID: \"7764fad6-5190-4fab-9952-7a2d9d226b9f\") " pod="openshift-console/console-54c8d7df74-vx8rr" Aug 04 16:00:48.708795 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:48.708679 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7764fad6-5190-4fab-9952-7a2d9d226b9f-service-ca\") pod \"console-54c8d7df74-vx8rr\" (UID: \"7764fad6-5190-4fab-9952-7a2d9d226b9f\") " pod="openshift-console/console-54c8d7df74-vx8rr" Aug 04 16:00:48.708795 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:48.708763 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7764fad6-5190-4fab-9952-7a2d9d226b9f-trusted-ca-bundle\") pod \"console-54c8d7df74-vx8rr\" (UID: \"7764fad6-5190-4fab-9952-7a2d9d226b9f\") " pod="openshift-console/console-54c8d7df74-vx8rr" Aug 04 16:00:48.708971 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:48.708806 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7764fad6-5190-4fab-9952-7a2d9d226b9f-console-oauth-config\") pod \"console-54c8d7df74-vx8rr\" (UID: \"7764fad6-5190-4fab-9952-7a2d9d226b9f\") " pod="openshift-console/console-54c8d7df74-vx8rr" Aug 04 16:00:48.708971 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:48.708842 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjzds\" (UniqueName: \"kubernetes.io/projected/7764fad6-5190-4fab-9952-7a2d9d226b9f-kube-api-access-rjzds\") pod \"console-54c8d7df74-vx8rr\" (UID: \"7764fad6-5190-4fab-9952-7a2d9d226b9f\") " pod="openshift-console/console-54c8d7df74-vx8rr" Aug 04 16:00:48.708971 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:48.708878 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7764fad6-5190-4fab-9952-7a2d9d226b9f-console-config\") pod \"console-54c8d7df74-vx8rr\" (UID: \"7764fad6-5190-4fab-9952-7a2d9d226b9f\") " pod="openshift-console/console-54c8d7df74-vx8rr" Aug 04 16:00:48.709440 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:48.709418 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7764fad6-5190-4fab-9952-7a2d9d226b9f-service-ca\") pod \"console-54c8d7df74-vx8rr\" (UID: \"7764fad6-5190-4fab-9952-7a2d9d226b9f\") " pod="openshift-console/console-54c8d7df74-vx8rr" Aug 04 16:00:48.709541 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:48.709416 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7764fad6-5190-4fab-9952-7a2d9d226b9f-oauth-serving-cert\") pod \"console-54c8d7df74-vx8rr\" (UID: \"7764fad6-5190-4fab-9952-7a2d9d226b9f\") " pod="openshift-console/console-54c8d7df74-vx8rr" Aug 04 16:00:48.709541 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:48.709529 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7764fad6-5190-4fab-9952-7a2d9d226b9f-trusted-ca-bundle\") pod \"console-54c8d7df74-vx8rr\" (UID: \"7764fad6-5190-4fab-9952-7a2d9d226b9f\") " pod="openshift-console/console-54c8d7df74-vx8rr" Aug 04 16:00:48.709680 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:48.709659 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7764fad6-5190-4fab-9952-7a2d9d226b9f-console-config\") pod \"console-54c8d7df74-vx8rr\" (UID: \"7764fad6-5190-4fab-9952-7a2d9d226b9f\") " pod="openshift-console/console-54c8d7df74-vx8rr" Aug 04 16:00:48.711093 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:48.711073 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7764fad6-5190-4fab-9952-7a2d9d226b9f-console-serving-cert\") pod \"console-54c8d7df74-vx8rr\" (UID: \"7764fad6-5190-4fab-9952-7a2d9d226b9f\") " pod="openshift-console/console-54c8d7df74-vx8rr" Aug 04 16:00:48.711193 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:48.711178 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7764fad6-5190-4fab-9952-7a2d9d226b9f-console-oauth-config\") pod \"console-54c8d7df74-vx8rr\" (UID: \"7764fad6-5190-4fab-9952-7a2d9d226b9f\") " pod="openshift-console/console-54c8d7df74-vx8rr" Aug 04 16:00:48.717504 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:48.717486 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjzds\" (UniqueName: \"kubernetes.io/projected/7764fad6-5190-4fab-9952-7a2d9d226b9f-kube-api-access-rjzds\") pod \"console-54c8d7df74-vx8rr\" (UID: \"7764fad6-5190-4fab-9952-7a2d9d226b9f\") " pod="openshift-console/console-54c8d7df74-vx8rr" Aug 04 16:00:48.824340 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:48.824256 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54c8d7df74-vx8rr" Aug 04 16:00:48.956380 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:48.956350 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54c8d7df74-vx8rr"] Aug 04 16:00:48.959342 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:00:48.959307 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7764fad6_5190_4fab_9952_7a2d9d226b9f.slice/crio-71bf7bbb982482e529981160ce61a9a2d6c718c10e0a9571e01247000fd6fe38 WatchSource:0}: Error finding container 71bf7bbb982482e529981160ce61a9a2d6c718c10e0a9571e01247000fd6fe38: Status 404 returned error can't find the container with id 71bf7bbb982482e529981160ce61a9a2d6c718c10e0a9571e01247000fd6fe38 Aug 04 16:00:49.000188 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:49.000155 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54c8d7df74-vx8rr" event={"ID":"7764fad6-5190-4fab-9952-7a2d9d226b9f","Type":"ContainerStarted","Data":"71bf7bbb982482e529981160ce61a9a2d6c718c10e0a9571e01247000fd6fe38"} Aug 04 16:00:54.020609 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:54.020571 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54c8d7df74-vx8rr" event={"ID":"7764fad6-5190-4fab-9952-7a2d9d226b9f","Type":"ContainerStarted","Data":"25e76bb292338019a0fb199747a1122bb7751eb0015d342ed32e35dab68f8191"} Aug 04 16:00:54.041161 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:54.041110 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54c8d7df74-vx8rr" podStartSLOduration=1.5997345630000002 podStartE2EDuration="6.041094518s" totalImagesPullingTime="4.441359955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:00:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:00:54.040062179 +0000 UTC m=+253.457604342" watchObservedRunningTime="2026-08-04 16:00:54.041094518 +0000 UTC m=+253.458636712" Aug 04 16:00:58.824766 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:58.824719 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54c8d7df74-vx8rr" Aug 04 16:00:58.825222 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:58.824792 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54c8d7df74-vx8rr" Aug 04 16:00:58.829417 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:58.829398 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54c8d7df74-vx8rr" Aug 04 16:00:59.040606 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:00:59.040580 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54c8d7df74-vx8rr" Aug 04 16:01:40.939473 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:01:40.939436 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-gvb4l_c16f82ab-8340-49ee-b41e-5685d4592fd2/console-operator/2.log" Aug 04 16:01:40.943489 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:01:40.943464 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mtbc5_11f9cbee-a773-420b-9873-5c8cb7679dd8/ovn-acl-logging/0.log" Aug 04 16:01:40.943489 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:01:40.943478 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-gvb4l_c16f82ab-8340-49ee-b41e-5685d4592fd2/console-operator/2.log" Aug 04 16:01:40.948460 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:01:40.948435 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mtbc5_11f9cbee-a773-420b-9873-5c8cb7679dd8/ovn-acl-logging/0.log" Aug 04 16:01:40.949934 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:01:40.949914 2573 kubelet.go:1648] "Image garbage collection succeeded" Aug 04 16:02:09.721383 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:02:09.721345 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54c8d7df74-vx8rr"] Aug 04 16:02:34.742145 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:02:34.742055 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-54c8d7df74-vx8rr" podUID="7764fad6-5190-4fab-9952-7a2d9d226b9f" containerName="console" containerID="cri-o://25e76bb292338019a0fb199747a1122bb7751eb0015d342ed32e35dab68f8191" gracePeriod=15 Aug 04 16:02:34.980594 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:02:34.980570 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54c8d7df74-vx8rr" Aug 04 16:02:35.073041 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:02:35.072951 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7764fad6-5190-4fab-9952-7a2d9d226b9f-console-config\") pod \"7764fad6-5190-4fab-9952-7a2d9d226b9f\" (UID: \"7764fad6-5190-4fab-9952-7a2d9d226b9f\") " Aug 04 16:02:35.073041 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:02:35.073003 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjzds\" (UniqueName: \"kubernetes.io/projected/7764fad6-5190-4fab-9952-7a2d9d226b9f-kube-api-access-rjzds\") pod \"7764fad6-5190-4fab-9952-7a2d9d226b9f\" (UID: \"7764fad6-5190-4fab-9952-7a2d9d226b9f\") " Aug 04 16:02:35.073041 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:02:35.073023 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7764fad6-5190-4fab-9952-7a2d9d226b9f-console-serving-cert\") pod \"7764fad6-5190-4fab-9952-7a2d9d226b9f\" (UID: \"7764fad6-5190-4fab-9952-7a2d9d226b9f\") " Aug 04 16:02:35.073041 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:02:35.073042 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7764fad6-5190-4fab-9952-7a2d9d226b9f-oauth-serving-cert\") pod \"7764fad6-5190-4fab-9952-7a2d9d226b9f\" (UID: \"7764fad6-5190-4fab-9952-7a2d9d226b9f\") " Aug 04 16:02:35.073372 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:02:35.073066 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7764fad6-5190-4fab-9952-7a2d9d226b9f-console-oauth-config\") pod \"7764fad6-5190-4fab-9952-7a2d9d226b9f\" (UID: \"7764fad6-5190-4fab-9952-7a2d9d226b9f\") " Aug 04 16:02:35.073372 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:02:35.073106 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7764fad6-5190-4fab-9952-7a2d9d226b9f-service-ca\") pod \"7764fad6-5190-4fab-9952-7a2d9d226b9f\" (UID: \"7764fad6-5190-4fab-9952-7a2d9d226b9f\") " Aug 04 16:02:35.073372 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:02:35.073141 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7764fad6-5190-4fab-9952-7a2d9d226b9f-trusted-ca-bundle\") pod \"7764fad6-5190-4fab-9952-7a2d9d226b9f\" (UID: \"7764fad6-5190-4fab-9952-7a2d9d226b9f\") " Aug 04 16:02:35.073535 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:02:35.073364 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7764fad6-5190-4fab-9952-7a2d9d226b9f-console-config" (OuterVolumeSpecName: "console-config") pod "7764fad6-5190-4fab-9952-7a2d9d226b9f" (UID: "7764fad6-5190-4fab-9952-7a2d9d226b9f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:02:35.073620 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:02:35.073526 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7764fad6-5190-4fab-9952-7a2d9d226b9f-service-ca" (OuterVolumeSpecName: "service-ca") pod "7764fad6-5190-4fab-9952-7a2d9d226b9f" (UID: "7764fad6-5190-4fab-9952-7a2d9d226b9f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:02:35.073620 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:02:35.073594 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7764fad6-5190-4fab-9952-7a2d9d226b9f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7764fad6-5190-4fab-9952-7a2d9d226b9f" (UID: "7764fad6-5190-4fab-9952-7a2d9d226b9f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:02:35.073722 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:02:35.073649 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7764fad6-5190-4fab-9952-7a2d9d226b9f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7764fad6-5190-4fab-9952-7a2d9d226b9f" (UID: "7764fad6-5190-4fab-9952-7a2d9d226b9f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:02:35.077595 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:02:35.075741 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7764fad6-5190-4fab-9952-7a2d9d226b9f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7764fad6-5190-4fab-9952-7a2d9d226b9f" (UID: "7764fad6-5190-4fab-9952-7a2d9d226b9f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:02:35.080173 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:02:35.079901 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7764fad6-5190-4fab-9952-7a2d9d226b9f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7764fad6-5190-4fab-9952-7a2d9d226b9f" (UID: "7764fad6-5190-4fab-9952-7a2d9d226b9f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:02:35.080687 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:02:35.080577 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7764fad6-5190-4fab-9952-7a2d9d226b9f-kube-api-access-rjzds" (OuterVolumeSpecName: "kube-api-access-rjzds") pod "7764fad6-5190-4fab-9952-7a2d9d226b9f" (UID: "7764fad6-5190-4fab-9952-7a2d9d226b9f"). InnerVolumeSpecName "kube-api-access-rjzds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:02:35.173721 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:02:35.173682 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjzds\" (UniqueName: \"kubernetes.io/projected/7764fad6-5190-4fab-9952-7a2d9d226b9f-kube-api-access-rjzds\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:02:35.173721 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:02:35.173710 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7764fad6-5190-4fab-9952-7a2d9d226b9f-console-serving-cert\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:02:35.173721 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:02:35.173720 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7764fad6-5190-4fab-9952-7a2d9d226b9f-oauth-serving-cert\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:02:35.173721 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:02:35.173728 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7764fad6-5190-4fab-9952-7a2d9d226b9f-console-oauth-config\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:02:35.173985 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:02:35.173737 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7764fad6-5190-4fab-9952-7a2d9d226b9f-service-ca\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:02:35.173985 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:02:35.173745 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7764fad6-5190-4fab-9952-7a2d9d226b9f-trusted-ca-bundle\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:02:35.173985 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:02:35.173753 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7764fad6-5190-4fab-9952-7a2d9d226b9f-console-config\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:02:35.362673 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:02:35.362581 2573 generic.go:356] "Generic (PLEG): container finished" podID="7764fad6-5190-4fab-9952-7a2d9d226b9f" containerID="25e76bb292338019a0fb199747a1122bb7751eb0015d342ed32e35dab68f8191" exitCode=0 Aug 04 16:02:35.362673 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:02:35.362624 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54c8d7df74-vx8rr" Aug 04 16:02:35.362852 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:02:35.362668 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54c8d7df74-vx8rr" event={"ID":"7764fad6-5190-4fab-9952-7a2d9d226b9f","Type":"ContainerDied","Data":"25e76bb292338019a0fb199747a1122bb7751eb0015d342ed32e35dab68f8191"} Aug 04 16:02:35.362852 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:02:35.362720 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54c8d7df74-vx8rr" event={"ID":"7764fad6-5190-4fab-9952-7a2d9d226b9f","Type":"ContainerDied","Data":"71bf7bbb982482e529981160ce61a9a2d6c718c10e0a9571e01247000fd6fe38"} Aug 04 16:02:35.362852 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:02:35.362744 2573 scope.go:117] "RemoveContainer" containerID="25e76bb292338019a0fb199747a1122bb7751eb0015d342ed32e35dab68f8191" Aug 04 16:02:35.370704 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:02:35.370694 2573 scope.go:117] "RemoveContainer" containerID="25e76bb292338019a0fb199747a1122bb7751eb0015d342ed32e35dab68f8191" Aug 04 16:02:35.370976 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:02:35.370953 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25e76bb292338019a0fb199747a1122bb7751eb0015d342ed32e35dab68f8191\": container with ID starting with 25e76bb292338019a0fb199747a1122bb7751eb0015d342ed32e35dab68f8191 not found: ID does not exist" containerID="25e76bb292338019a0fb199747a1122bb7751eb0015d342ed32e35dab68f8191" Aug 04 16:02:35.371048 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:02:35.370986 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25e76bb292338019a0fb199747a1122bb7751eb0015d342ed32e35dab68f8191"} err="failed to get container status \"25e76bb292338019a0fb199747a1122bb7751eb0015d342ed32e35dab68f8191\": rpc error: code = NotFound desc = could not find container \"25e76bb292338019a0fb199747a1122bb7751eb0015d342ed32e35dab68f8191\": container with ID starting with 25e76bb292338019a0fb199747a1122bb7751eb0015d342ed32e35dab68f8191 not found: ID does not exist" Aug 04 16:02:35.378288 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:02:35.378266 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54c8d7df74-vx8rr"] Aug 04 16:02:35.382031 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:02:35.382011 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54c8d7df74-vx8rr"] Aug 04 16:02:37.077005 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:02:37.076974 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7764fad6-5190-4fab-9952-7a2d9d226b9f" path="/var/lib/kubelet/pods/7764fad6-5190-4fab-9952-7a2d9d226b9f/volumes" Aug 04 16:05:31.837115 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:31.837080 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-q5rgb"] Aug 04 16:05:31.837633 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:31.837320 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7764fad6-5190-4fab-9952-7a2d9d226b9f" containerName="console" Aug 04 16:05:31.837633 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:31.837332 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7764fad6-5190-4fab-9952-7a2d9d226b9f" containerName="console" Aug 04 16:05:31.837633 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:31.837374 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7764fad6-5190-4fab-9952-7a2d9d226b9f" containerName="console" Aug 04 16:05:31.839748 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:31.839733 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-q5rgb" Aug 04 16:05:31.842638 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:31.842614 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Aug 04 16:05:31.842783 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:31.842672 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Aug 04 16:05:31.843064 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:31.843051 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-jmxzp\"" Aug 04 16:05:31.852544 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:31.852520 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-q5rgb"] Aug 04 16:05:31.930435 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:31.930400 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnhhm\" (UniqueName: \"kubernetes.io/projected/1fa82a3b-e179-4f81-a4ac-192c66c05074-kube-api-access-lnhhm\") pod \"cert-manager-55b4d954d5-q5rgb\" (UID: \"1fa82a3b-e179-4f81-a4ac-192c66c05074\") " pod="cert-manager/cert-manager-55b4d954d5-q5rgb" Aug 04 16:05:31.930648 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:31.930458 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fa82a3b-e179-4f81-a4ac-192c66c05074-bound-sa-token\") pod \"cert-manager-55b4d954d5-q5rgb\" (UID: \"1fa82a3b-e179-4f81-a4ac-192c66c05074\") " pod="cert-manager/cert-manager-55b4d954d5-q5rgb" Aug 04 16:05:32.031855 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:32.031809 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fa82a3b-e179-4f81-a4ac-192c66c05074-bound-sa-token\") pod \"cert-manager-55b4d954d5-q5rgb\" (UID: \"1fa82a3b-e179-4f81-a4ac-192c66c05074\") " pod="cert-manager/cert-manager-55b4d954d5-q5rgb" Aug 04 16:05:32.032073 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:32.031876 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnhhm\" (UniqueName: \"kubernetes.io/projected/1fa82a3b-e179-4f81-a4ac-192c66c05074-kube-api-access-lnhhm\") pod \"cert-manager-55b4d954d5-q5rgb\" (UID: \"1fa82a3b-e179-4f81-a4ac-192c66c05074\") " pod="cert-manager/cert-manager-55b4d954d5-q5rgb" Aug 04 16:05:32.040889 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:32.040862 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fa82a3b-e179-4f81-a4ac-192c66c05074-bound-sa-token\") pod \"cert-manager-55b4d954d5-q5rgb\" (UID: \"1fa82a3b-e179-4f81-a4ac-192c66c05074\") " pod="cert-manager/cert-manager-55b4d954d5-q5rgb" Aug 04 16:05:32.043190 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:32.043169 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnhhm\" (UniqueName: \"kubernetes.io/projected/1fa82a3b-e179-4f81-a4ac-192c66c05074-kube-api-access-lnhhm\") pod \"cert-manager-55b4d954d5-q5rgb\" (UID: \"1fa82a3b-e179-4f81-a4ac-192c66c05074\") " pod="cert-manager/cert-manager-55b4d954d5-q5rgb" Aug 04 16:05:32.148013 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:32.147928 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-q5rgb" Aug 04 16:05:32.278841 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:32.278799 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-q5rgb"] Aug 04 16:05:32.286295 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:32.286274 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 16:05:32.961693 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:32.961656 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-q5rgb" event={"ID":"1fa82a3b-e179-4f81-a4ac-192c66c05074","Type":"ContainerStarted","Data":"31b2ad278d4a34e67d4bc23b6a2f1d5b5065096ed140ee3890eb67c6c7720e4d"} Aug 04 16:05:35.974792 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:35.974751 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-q5rgb" event={"ID":"1fa82a3b-e179-4f81-a4ac-192c66c05074","Type":"ContainerStarted","Data":"28018f1d092948b39528cd153331a09e40d8eb342e8d82e71c74b8b87ae9fbd4"} Aug 04 16:05:35.994572 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:35.994509 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-q5rgb" podStartSLOduration=2.038738153 podStartE2EDuration="4.994492685s" totalImagesPullingTime="2.955754532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:05:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:05:35.992989325 +0000 UTC m=+535.410531452" watchObservedRunningTime="2026-08-04 16:05:35.994492685 +0000 UTC m=+535.412034810" Aug 04 16:05:58.385064 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:58.385027 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7dfb598967-6cw62"] Aug 04 16:05:58.389489 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:58.389473 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-6cw62" Aug 04 16:05:58.392066 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:58.392043 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-ft4sg\"" Aug 04 16:05:58.392066 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:58.392055 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Aug 04 16:05:58.392301 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:58.392288 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Aug 04 16:05:58.392731 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:58.392715 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Aug 04 16:05:58.392821 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:58.392717 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Aug 04 16:05:58.417358 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:58.417323 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7dfb598967-6cw62"] Aug 04 16:05:58.430526 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:58.430497 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3de24931-7b30-4df7-8f31-6edef899c2a2-manifests\") pod \"opendatahub-operator-controller-manager-7dfb598967-6cw62\" (UID: \"3de24931-7b30-4df7-8f31-6edef899c2a2\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-6cw62" Aug 04 16:05:58.430678 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:58.430532 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnqbl\" (UniqueName: \"kubernetes.io/projected/3de24931-7b30-4df7-8f31-6edef899c2a2-kube-api-access-jnqbl\") pod \"opendatahub-operator-controller-manager-7dfb598967-6cw62\" (UID: \"3de24931-7b30-4df7-8f31-6edef899c2a2\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-6cw62" Aug 04 16:05:58.430678 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:58.430591 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3de24931-7b30-4df7-8f31-6edef899c2a2-tmp\") pod \"opendatahub-operator-controller-manager-7dfb598967-6cw62\" (UID: \"3de24931-7b30-4df7-8f31-6edef899c2a2\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-6cw62" Aug 04 16:05:58.430678 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:58.430635 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3de24931-7b30-4df7-8f31-6edef899c2a2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7dfb598967-6cw62\" (UID: \"3de24931-7b30-4df7-8f31-6edef899c2a2\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-6cw62" Aug 04 16:05:58.430781 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:58.430678 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3de24931-7b30-4df7-8f31-6edef899c2a2-webhook-cert\") pod \"opendatahub-operator-controller-manager-7dfb598967-6cw62\" (UID: \"3de24931-7b30-4df7-8f31-6edef899c2a2\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-6cw62" Aug 04 16:05:58.531171 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:58.531138 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3de24931-7b30-4df7-8f31-6edef899c2a2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7dfb598967-6cw62\" (UID: \"3de24931-7b30-4df7-8f31-6edef899c2a2\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-6cw62" Aug 04 16:05:58.531171 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:58.531168 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3de24931-7b30-4df7-8f31-6edef899c2a2-webhook-cert\") pod \"opendatahub-operator-controller-manager-7dfb598967-6cw62\" (UID: \"3de24931-7b30-4df7-8f31-6edef899c2a2\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-6cw62" Aug 04 16:05:58.531411 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:58.531194 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3de24931-7b30-4df7-8f31-6edef899c2a2-manifests\") pod \"opendatahub-operator-controller-manager-7dfb598967-6cw62\" (UID: \"3de24931-7b30-4df7-8f31-6edef899c2a2\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-6cw62" Aug 04 16:05:58.531411 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:58.531211 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnqbl\" (UniqueName: \"kubernetes.io/projected/3de24931-7b30-4df7-8f31-6edef899c2a2-kube-api-access-jnqbl\") pod \"opendatahub-operator-controller-manager-7dfb598967-6cw62\" (UID: \"3de24931-7b30-4df7-8f31-6edef899c2a2\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-6cw62" Aug 04 16:05:58.531411 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:58.531337 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3de24931-7b30-4df7-8f31-6edef899c2a2-tmp\") pod \"opendatahub-operator-controller-manager-7dfb598967-6cw62\" (UID: \"3de24931-7b30-4df7-8f31-6edef899c2a2\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-6cw62" Aug 04 16:05:58.531656 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:58.531637 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3de24931-7b30-4df7-8f31-6edef899c2a2-manifests\") pod \"opendatahub-operator-controller-manager-7dfb598967-6cw62\" (UID: \"3de24931-7b30-4df7-8f31-6edef899c2a2\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-6cw62" Aug 04 16:05:58.533472 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:58.533452 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3de24931-7b30-4df7-8f31-6edef899c2a2-tmp\") pod \"opendatahub-operator-controller-manager-7dfb598967-6cw62\" (UID: \"3de24931-7b30-4df7-8f31-6edef899c2a2\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-6cw62" Aug 04 16:05:58.533637 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:58.533619 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3de24931-7b30-4df7-8f31-6edef899c2a2-webhook-cert\") pod \"opendatahub-operator-controller-manager-7dfb598967-6cw62\" (UID: \"3de24931-7b30-4df7-8f31-6edef899c2a2\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-6cw62" Aug 04 16:05:58.533706 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:58.533684 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3de24931-7b30-4df7-8f31-6edef899c2a2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7dfb598967-6cw62\" (UID: \"3de24931-7b30-4df7-8f31-6edef899c2a2\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-6cw62" Aug 04 16:05:58.548664 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:58.548635 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnqbl\" (UniqueName: \"kubernetes.io/projected/3de24931-7b30-4df7-8f31-6edef899c2a2-kube-api-access-jnqbl\") pod \"opendatahub-operator-controller-manager-7dfb598967-6cw62\" (UID: \"3de24931-7b30-4df7-8f31-6edef899c2a2\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-6cw62" Aug 04 16:05:58.699752 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:58.699725 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-6cw62" Aug 04 16:05:58.837910 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:58.837878 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7dfb598967-6cw62"] Aug 04 16:05:58.843262 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:05:58.843236 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de24931_7b30_4df7_8f31_6edef899c2a2.slice/crio-bc4c9e6431d4061a11905d0beeb25a27021d5fa14190e3d4382e0f1f18dcf228 WatchSource:0}: Error finding container bc4c9e6431d4061a11905d0beeb25a27021d5fa14190e3d4382e0f1f18dcf228: Status 404 returned error can't find the container with id bc4c9e6431d4061a11905d0beeb25a27021d5fa14190e3d4382e0f1f18dcf228 Aug 04 16:05:59.058865 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:05:59.058776 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-6cw62" event={"ID":"3de24931-7b30-4df7-8f31-6edef899c2a2","Type":"ContainerStarted","Data":"bc4c9e6431d4061a11905d0beeb25a27021d5fa14190e3d4382e0f1f18dcf228"} Aug 04 16:06:04.082034 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:04.081932 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-6cw62" event={"ID":"3de24931-7b30-4df7-8f31-6edef899c2a2","Type":"ContainerStarted","Data":"ba54dd8a478f568e3f82e919970a9ff4afa67f13e9f8abe7a0fc6a6170147f4c"} Aug 04 16:06:05.086305 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:05.086270 2573 generic.go:356] "Generic (PLEG): container finished" podID="3de24931-7b30-4df7-8f31-6edef899c2a2" containerID="ba54dd8a478f568e3f82e919970a9ff4afa67f13e9f8abe7a0fc6a6170147f4c" exitCode=0 Aug 04 16:06:05.086692 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:05.086316 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-6cw62" event={"ID":"3de24931-7b30-4df7-8f31-6edef899c2a2","Type":"ContainerDied","Data":"ba54dd8a478f568e3f82e919970a9ff4afa67f13e9f8abe7a0fc6a6170147f4c"} Aug 04 16:06:06.091819 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:06.091779 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-6cw62" event={"ID":"3de24931-7b30-4df7-8f31-6edef899c2a2","Type":"ContainerStarted","Data":"c610180e6f0b2a29fcf438e8e68adfdfba6700e8f7a1de3cd118bb0f8dd9f491"} Aug 04 16:06:06.092190 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:06.091927 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-6cw62" Aug 04 16:06:06.113118 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:06.113065 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-6cw62" podStartSLOduration=1.642207259 podStartE2EDuration="8.113049111s" totalImagesPullingTime="5.399222364s" totalInitContainerRuntime="1.071619488s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:05:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:06:06.111086409 +0000 UTC m=+565.528628529" watchObservedRunningTime="2026-08-04 16:06:06.113049111 +0000 UTC m=+565.530591237" Aug 04 16:06:17.097866 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:17.097836 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-6cw62" Aug 04 16:06:20.347095 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:20.347058 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-78b5c5f4c4-n5rwh"] Aug 04 16:06:20.355362 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:20.355333 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-78b5c5f4c4-n5rwh" Aug 04 16:06:20.358423 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:20.358400 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Aug 04 16:06:20.359089 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:20.359068 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-5zk85\"" Aug 04 16:06:20.362701 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:20.362681 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Aug 04 16:06:20.377104 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:20.377078 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-78b5c5f4c4-n5rwh"] Aug 04 16:06:20.383142 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:20.383119 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4csl\" (UniqueName: \"kubernetes.io/projected/472fadc0-3396-40f3-8d6d-1e93966ec942-kube-api-access-g4csl\") pod \"kube-auth-proxy-78b5c5f4c4-n5rwh\" (UID: \"472fadc0-3396-40f3-8d6d-1e93966ec942\") " pod="openshift-ingress/kube-auth-proxy-78b5c5f4c4-n5rwh" Aug 04 16:06:20.383262 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:20.383152 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/472fadc0-3396-40f3-8d6d-1e93966ec942-tmp\") pod \"kube-auth-proxy-78b5c5f4c4-n5rwh\" (UID: \"472fadc0-3396-40f3-8d6d-1e93966ec942\") " pod="openshift-ingress/kube-auth-proxy-78b5c5f4c4-n5rwh" Aug 04 16:06:20.383262 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:20.383183 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/472fadc0-3396-40f3-8d6d-1e93966ec942-tls-certs\") pod \"kube-auth-proxy-78b5c5f4c4-n5rwh\" (UID: \"472fadc0-3396-40f3-8d6d-1e93966ec942\") " pod="openshift-ingress/kube-auth-proxy-78b5c5f4c4-n5rwh" Aug 04 16:06:20.484374 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:20.484322 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4csl\" (UniqueName: \"kubernetes.io/projected/472fadc0-3396-40f3-8d6d-1e93966ec942-kube-api-access-g4csl\") pod \"kube-auth-proxy-78b5c5f4c4-n5rwh\" (UID: \"472fadc0-3396-40f3-8d6d-1e93966ec942\") " pod="openshift-ingress/kube-auth-proxy-78b5c5f4c4-n5rwh" Aug 04 16:06:20.484374 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:20.484379 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/472fadc0-3396-40f3-8d6d-1e93966ec942-tmp\") pod \"kube-auth-proxy-78b5c5f4c4-n5rwh\" (UID: \"472fadc0-3396-40f3-8d6d-1e93966ec942\") " pod="openshift-ingress/kube-auth-proxy-78b5c5f4c4-n5rwh" Aug 04 16:06:20.484699 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:20.484422 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/472fadc0-3396-40f3-8d6d-1e93966ec942-tls-certs\") pod \"kube-auth-proxy-78b5c5f4c4-n5rwh\" (UID: \"472fadc0-3396-40f3-8d6d-1e93966ec942\") " pod="openshift-ingress/kube-auth-proxy-78b5c5f4c4-n5rwh" Aug 04 16:06:20.484699 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:06:20.484537 2573 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Aug 04 16:06:20.484699 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:06:20.484627 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/472fadc0-3396-40f3-8d6d-1e93966ec942-tls-certs podName:472fadc0-3396-40f3-8d6d-1e93966ec942 nodeName:}" failed. No retries permitted until 2026-08-04 16:06:20.984607269 +0000 UTC m=+580.402149373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/472fadc0-3396-40f3-8d6d-1e93966ec942-tls-certs") pod "kube-auth-proxy-78b5c5f4c4-n5rwh" (UID: "472fadc0-3396-40f3-8d6d-1e93966ec942") : secret "kube-auth-proxy-tls" not found Aug 04 16:06:20.486872 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:20.486841 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/472fadc0-3396-40f3-8d6d-1e93966ec942-tmp\") pod \"kube-auth-proxy-78b5c5f4c4-n5rwh\" (UID: \"472fadc0-3396-40f3-8d6d-1e93966ec942\") " pod="openshift-ingress/kube-auth-proxy-78b5c5f4c4-n5rwh" Aug 04 16:06:20.524476 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:20.524447 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4csl\" (UniqueName: \"kubernetes.io/projected/472fadc0-3396-40f3-8d6d-1e93966ec942-kube-api-access-g4csl\") pod \"kube-auth-proxy-78b5c5f4c4-n5rwh\" (UID: \"472fadc0-3396-40f3-8d6d-1e93966ec942\") " pod="openshift-ingress/kube-auth-proxy-78b5c5f4c4-n5rwh" Aug 04 16:06:20.989216 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:20.989184 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/472fadc0-3396-40f3-8d6d-1e93966ec942-tls-certs\") pod \"kube-auth-proxy-78b5c5f4c4-n5rwh\" (UID: \"472fadc0-3396-40f3-8d6d-1e93966ec942\") " pod="openshift-ingress/kube-auth-proxy-78b5c5f4c4-n5rwh" Aug 04 16:06:20.991796 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:20.991768 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/472fadc0-3396-40f3-8d6d-1e93966ec942-tls-certs\") pod \"kube-auth-proxy-78b5c5f4c4-n5rwh\" (UID: \"472fadc0-3396-40f3-8d6d-1e93966ec942\") " pod="openshift-ingress/kube-auth-proxy-78b5c5f4c4-n5rwh" Aug 04 16:06:21.264355 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:21.264267 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-78b5c5f4c4-n5rwh" Aug 04 16:06:21.432135 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:21.432089 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-78b5c5f4c4-n5rwh"] Aug 04 16:06:21.434798 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:06:21.434765 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod472fadc0_3396_40f3_8d6d_1e93966ec942.slice/crio-f730450a6f34109151fc542ec473a95c6ca99adfea9de55710fc7af0b498e9d1 WatchSource:0}: Error finding container f730450a6f34109151fc542ec473a95c6ca99adfea9de55710fc7af0b498e9d1: Status 404 returned error can't find the container with id f730450a6f34109151fc542ec473a95c6ca99adfea9de55710fc7af0b498e9d1 Aug 04 16:06:22.148681 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:22.148638 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-78b5c5f4c4-n5rwh" event={"ID":"472fadc0-3396-40f3-8d6d-1e93966ec942","Type":"ContainerStarted","Data":"f730450a6f34109151fc542ec473a95c6ca99adfea9de55710fc7af0b498e9d1"} Aug 04 16:06:22.506270 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:22.506229 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5"] Aug 04 16:06:22.509633 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:22.509607 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5" Aug 04 16:06:22.513429 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:22.513407 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Aug 04 16:06:22.513589 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:22.513447 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Aug 04 16:06:22.513740 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:22.513720 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Aug 04 16:06:22.513883 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:22.513766 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Aug 04 16:06:22.514152 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:22.514136 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4tx97\"" Aug 04 16:06:22.523709 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:22.523685 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5"] Aug 04 16:06:22.601238 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:22.601171 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/55f89be8-4514-49a5-9192-33a6d150864d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-lnsx5\" (UID: \"55f89be8-4514-49a5-9192-33a6d150864d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5" Aug 04 16:06:22.601238 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:22.601218 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/55f89be8-4514-49a5-9192-33a6d150864d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-lnsx5\" (UID: \"55f89be8-4514-49a5-9192-33a6d150864d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5" Aug 04 16:06:22.601492 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:22.601298 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zmrd\" (UniqueName: \"kubernetes.io/projected/55f89be8-4514-49a5-9192-33a6d150864d-kube-api-access-9zmrd\") pod \"istiod-openshift-gateway-7b7995d5cd-lnsx5\" (UID: \"55f89be8-4514-49a5-9192-33a6d150864d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5" Aug 04 16:06:22.601492 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:22.601326 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/55f89be8-4514-49a5-9192-33a6d150864d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-lnsx5\" (UID: \"55f89be8-4514-49a5-9192-33a6d150864d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5" Aug 04 16:06:22.601492 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:22.601348 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/55f89be8-4514-49a5-9192-33a6d150864d-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-lnsx5\" (UID: \"55f89be8-4514-49a5-9192-33a6d150864d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5" Aug 04 16:06:22.601492 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:22.601407 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f89be8-4514-49a5-9192-33a6d150864d-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-lnsx5\" (UID: \"55f89be8-4514-49a5-9192-33a6d150864d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5" Aug 04 16:06:22.601492 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:22.601472 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f89be8-4514-49a5-9192-33a6d150864d-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-lnsx5\" (UID: \"55f89be8-4514-49a5-9192-33a6d150864d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5" Aug 04 16:06:22.702258 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:22.702150 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f89be8-4514-49a5-9192-33a6d150864d-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-lnsx5\" (UID: \"55f89be8-4514-49a5-9192-33a6d150864d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5" Aug 04 16:06:22.702258 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:22.702214 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f89be8-4514-49a5-9192-33a6d150864d-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-lnsx5\" (UID: \"55f89be8-4514-49a5-9192-33a6d150864d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5" Aug 04 16:06:22.702258 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:22.702247 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/55f89be8-4514-49a5-9192-33a6d150864d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-lnsx5\" (UID: \"55f89be8-4514-49a5-9192-33a6d150864d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5" Aug 04 16:06:22.702523 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:22.702270 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/55f89be8-4514-49a5-9192-33a6d150864d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-lnsx5\" (UID: \"55f89be8-4514-49a5-9192-33a6d150864d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5" Aug 04 16:06:22.703304 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:22.702823 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zmrd\" (UniqueName: \"kubernetes.io/projected/55f89be8-4514-49a5-9192-33a6d150864d-kube-api-access-9zmrd\") pod \"istiod-openshift-gateway-7b7995d5cd-lnsx5\" (UID: \"55f89be8-4514-49a5-9192-33a6d150864d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5" Aug 04 16:06:22.703304 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:22.702869 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/55f89be8-4514-49a5-9192-33a6d150864d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-lnsx5\" (UID: \"55f89be8-4514-49a5-9192-33a6d150864d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5" Aug 04 16:06:22.703304 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:22.702897 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/55f89be8-4514-49a5-9192-33a6d150864d-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-lnsx5\" (UID: \"55f89be8-4514-49a5-9192-33a6d150864d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5" Aug 04 16:06:22.703676 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:22.703645 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/55f89be8-4514-49a5-9192-33a6d150864d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-lnsx5\" (UID: \"55f89be8-4514-49a5-9192-33a6d150864d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5" Aug 04 16:06:22.705358 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:22.705320 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f89be8-4514-49a5-9192-33a6d150864d-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-lnsx5\" (UID: \"55f89be8-4514-49a5-9192-33a6d150864d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5" Aug 04 16:06:22.705667 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:22.705644 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/55f89be8-4514-49a5-9192-33a6d150864d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-lnsx5\" (UID: \"55f89be8-4514-49a5-9192-33a6d150864d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5" Aug 04 16:06:22.705873 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:22.705848 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/55f89be8-4514-49a5-9192-33a6d150864d-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-lnsx5\" (UID: \"55f89be8-4514-49a5-9192-33a6d150864d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5" Aug 04 16:06:22.706010 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:22.705990 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/55f89be8-4514-49a5-9192-33a6d150864d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-lnsx5\" (UID: \"55f89be8-4514-49a5-9192-33a6d150864d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5" Aug 04 16:06:22.715623 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:22.715566 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zmrd\" (UniqueName: \"kubernetes.io/projected/55f89be8-4514-49a5-9192-33a6d150864d-kube-api-access-9zmrd\") pod \"istiod-openshift-gateway-7b7995d5cd-lnsx5\" (UID: \"55f89be8-4514-49a5-9192-33a6d150864d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5" Aug 04 16:06:22.715778 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:22.715653 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f89be8-4514-49a5-9192-33a6d150864d-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-lnsx5\" (UID: \"55f89be8-4514-49a5-9192-33a6d150864d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5" Aug 04 16:06:22.820827 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:22.820735 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5" Aug 04 16:06:22.992716 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:22.992673 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5"] Aug 04 16:06:22.997328 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:06:22.997295 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f89be8_4514_49a5_9192_33a6d150864d.slice/crio-ad00bd528024d0098ce5eb0eb66cd44eb72c978a18422154a7845e039b3af0b4 WatchSource:0}: Error finding container ad00bd528024d0098ce5eb0eb66cd44eb72c978a18422154a7845e039b3af0b4: Status 404 returned error can't find the container with id ad00bd528024d0098ce5eb0eb66cd44eb72c978a18422154a7845e039b3af0b4 Aug 04 16:06:23.155099 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:23.154980 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5" event={"ID":"55f89be8-4514-49a5-9192-33a6d150864d","Type":"ContainerStarted","Data":"ad00bd528024d0098ce5eb0eb66cd44eb72c978a18422154a7845e039b3af0b4"} Aug 04 16:06:25.165740 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:25.165710 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-78b5c5f4c4-n5rwh" event={"ID":"472fadc0-3396-40f3-8d6d-1e93966ec942","Type":"ContainerStarted","Data":"6f89a6593165f8464a763c4d645c9e3ef45e243073806f79ed080c8f0866a19e"} Aug 04 16:06:25.185919 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:25.185856 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-78b5c5f4c4-n5rwh" podStartSLOduration=2.00502018 podStartE2EDuration="5.185832622s" totalImagesPullingTime="3.180812442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:06:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:06:25.1843842 +0000 UTC m=+584.601926327" watchObservedRunningTime="2026-08-04 16:06:25.185832622 +0000 UTC m=+584.603374793" Aug 04 16:06:27.533931 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:27.533888 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Aug 04 16:06:27.534176 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:27.533965 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Aug 04 16:06:28.178950 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:28.178915 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5" event={"ID":"55f89be8-4514-49a5-9192-33a6d150864d","Type":"ContainerStarted","Data":"884a0ad4a6d80e8b7d248a79bb0477e35ef61f7a2177de5b1d97ed136fb523aa"} Aug 04 16:06:28.179161 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:28.179130 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5" Aug 04 16:06:28.180904 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:28.180877 2573 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-lnsx5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 16:06:28.181045 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:28.180941 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5" podUID="55f89be8-4514-49a5-9192-33a6d150864d" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 16:06:28.229159 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:28.229108 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5" podStartSLOduration=1.696108025 podStartE2EDuration="6.229093907s" totalImagesPullingTime="4.532985882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:06:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:06:28.227803476 +0000 UTC m=+587.645345596" watchObservedRunningTime="2026-08-04 16:06:28.229093907 +0000 UTC m=+587.646636032" Aug 04 16:06:29.183535 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:29.183501 2573 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-lnsx5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 16:06:29.183958 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:29.183588 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5" podUID="55f89be8-4514-49a5-9192-33a6d150864d" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 16:06:29.609362 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:29.609327 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5967759b96-fkvhc"] Aug 04 16:06:29.612668 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:29.612650 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5967759b96-fkvhc" Aug 04 16:06:29.619503 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:29.619483 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Aug 04 16:06:29.620914 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:29.620889 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Aug 04 16:06:29.621031 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:29.620989 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Aug 04 16:06:29.621100 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:29.621037 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-l5rrd\"" Aug 04 16:06:29.621207 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:29.621182 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Aug 04 16:06:29.621322 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:29.621248 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Aug 04 16:06:29.633945 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:29.633914 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5967759b96-fkvhc"] Aug 04 16:06:29.659496 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:29.659461 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl5d6\" (UniqueName: \"kubernetes.io/projected/0964e711-82e9-4d9f-a0b9-054a8429d92b-kube-api-access-sl5d6\") pod \"lws-controller-manager-5967759b96-fkvhc\" (UID: \"0964e711-82e9-4d9f-a0b9-054a8429d92b\") " pod="openshift-lws-operator/lws-controller-manager-5967759b96-fkvhc" Aug 04 16:06:29.659687 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:29.659505 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0964e711-82e9-4d9f-a0b9-054a8429d92b-manager-config\") pod \"lws-controller-manager-5967759b96-fkvhc\" (UID: \"0964e711-82e9-4d9f-a0b9-054a8429d92b\") " pod="openshift-lws-operator/lws-controller-manager-5967759b96-fkvhc" Aug 04 16:06:29.659687 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:29.659533 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0964e711-82e9-4d9f-a0b9-054a8429d92b-metrics-cert\") pod \"lws-controller-manager-5967759b96-fkvhc\" (UID: \"0964e711-82e9-4d9f-a0b9-054a8429d92b\") " pod="openshift-lws-operator/lws-controller-manager-5967759b96-fkvhc" Aug 04 16:06:29.659687 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:29.659640 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0964e711-82e9-4d9f-a0b9-054a8429d92b-cert\") pod \"lws-controller-manager-5967759b96-fkvhc\" (UID: \"0964e711-82e9-4d9f-a0b9-054a8429d92b\") " pod="openshift-lws-operator/lws-controller-manager-5967759b96-fkvhc" Aug 04 16:06:29.760508 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:29.760468 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl5d6\" (UniqueName: \"kubernetes.io/projected/0964e711-82e9-4d9f-a0b9-054a8429d92b-kube-api-access-sl5d6\") pod \"lws-controller-manager-5967759b96-fkvhc\" (UID: \"0964e711-82e9-4d9f-a0b9-054a8429d92b\") " pod="openshift-lws-operator/lws-controller-manager-5967759b96-fkvhc" Aug 04 16:06:29.760715 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:29.760525 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0964e711-82e9-4d9f-a0b9-054a8429d92b-manager-config\") pod \"lws-controller-manager-5967759b96-fkvhc\" (UID: \"0964e711-82e9-4d9f-a0b9-054a8429d92b\") " pod="openshift-lws-operator/lws-controller-manager-5967759b96-fkvhc" Aug 04 16:06:29.760715 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:29.760583 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0964e711-82e9-4d9f-a0b9-054a8429d92b-metrics-cert\") pod \"lws-controller-manager-5967759b96-fkvhc\" (UID: \"0964e711-82e9-4d9f-a0b9-054a8429d92b\") " pod="openshift-lws-operator/lws-controller-manager-5967759b96-fkvhc" Aug 04 16:06:29.760715 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:29.760624 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0964e711-82e9-4d9f-a0b9-054a8429d92b-cert\") pod \"lws-controller-manager-5967759b96-fkvhc\" (UID: \"0964e711-82e9-4d9f-a0b9-054a8429d92b\") " pod="openshift-lws-operator/lws-controller-manager-5967759b96-fkvhc" Aug 04 16:06:29.761747 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:29.761693 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0964e711-82e9-4d9f-a0b9-054a8429d92b-manager-config\") pod \"lws-controller-manager-5967759b96-fkvhc\" (UID: \"0964e711-82e9-4d9f-a0b9-054a8429d92b\") " pod="openshift-lws-operator/lws-controller-manager-5967759b96-fkvhc" Aug 04 16:06:29.764041 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:29.764011 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0964e711-82e9-4d9f-a0b9-054a8429d92b-cert\") pod \"lws-controller-manager-5967759b96-fkvhc\" (UID: \"0964e711-82e9-4d9f-a0b9-054a8429d92b\") " pod="openshift-lws-operator/lws-controller-manager-5967759b96-fkvhc" Aug 04 16:06:29.764201 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:29.764180 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0964e711-82e9-4d9f-a0b9-054a8429d92b-metrics-cert\") pod \"lws-controller-manager-5967759b96-fkvhc\" (UID: \"0964e711-82e9-4d9f-a0b9-054a8429d92b\") " pod="openshift-lws-operator/lws-controller-manager-5967759b96-fkvhc" Aug 04 16:06:29.773773 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:29.773743 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl5d6\" (UniqueName: \"kubernetes.io/projected/0964e711-82e9-4d9f-a0b9-054a8429d92b-kube-api-access-sl5d6\") pod \"lws-controller-manager-5967759b96-fkvhc\" (UID: \"0964e711-82e9-4d9f-a0b9-054a8429d92b\") " pod="openshift-lws-operator/lws-controller-manager-5967759b96-fkvhc" Aug 04 16:06:29.921882 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:29.921801 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5967759b96-fkvhc" Aug 04 16:06:30.083610 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:06:30.083573 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0964e711_82e9_4d9f_a0b9_054a8429d92b.slice/crio-4d5b556a9a1a01c39d40585f5e8f875be08e4c1e00981c089bf9e39b8782b7b6 WatchSource:0}: Error finding container 4d5b556a9a1a01c39d40585f5e8f875be08e4c1e00981c089bf9e39b8782b7b6: Status 404 returned error can't find the container with id 4d5b556a9a1a01c39d40585f5e8f875be08e4c1e00981c089bf9e39b8782b7b6 Aug 04 16:06:30.084979 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:30.084954 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5967759b96-fkvhc"] Aug 04 16:06:30.187919 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:30.187847 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5967759b96-fkvhc" event={"ID":"0964e711-82e9-4d9f-a0b9-054a8429d92b","Type":"ContainerStarted","Data":"4d5b556a9a1a01c39d40585f5e8f875be08e4c1e00981c089bf9e39b8782b7b6"} Aug 04 16:06:32.184199 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:32.184169 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5" Aug 04 16:06:33.201770 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:33.201733 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5967759b96-fkvhc" event={"ID":"0964e711-82e9-4d9f-a0b9-054a8429d92b","Type":"ContainerStarted","Data":"068e5eb05a444a2ce518d34bf5d0d4c9ca0ce96c63cf3684155cda6e26e6c465"} Aug 04 16:06:33.202299 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:33.201870 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5967759b96-fkvhc" Aug 04 16:06:33.297780 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:33.297731 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5967759b96-fkvhc" podStartSLOduration=1.634899158 podStartE2EDuration="4.297712584s" totalImagesPullingTime="2.662813426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:06:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:06:33.294068285 +0000 UTC m=+592.711610422" watchObservedRunningTime="2026-08-04 16:06:33.297712584 +0000 UTC m=+592.715254710" Aug 04 16:06:33.958445 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:33.958412 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5"] Aug 04 16:06:33.958639 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:33.958600 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5" podUID="55f89be8-4514-49a5-9192-33a6d150864d" containerName="discovery" containerID="cri-o://884a0ad4a6d80e8b7d248a79bb0477e35ef61f7a2177de5b1d97ed136fb523aa" gracePeriod=30 Aug 04 16:06:34.202854 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:34.202829 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5" Aug 04 16:06:34.206593 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:34.206570 2573 generic.go:356] "Generic (PLEG): container finished" podID="55f89be8-4514-49a5-9192-33a6d150864d" containerID="884a0ad4a6d80e8b7d248a79bb0477e35ef61f7a2177de5b1d97ed136fb523aa" exitCode=0 Aug 04 16:06:34.206714 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:34.206620 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5" Aug 04 16:06:34.206714 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:34.206652 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5" event={"ID":"55f89be8-4514-49a5-9192-33a6d150864d","Type":"ContainerDied","Data":"884a0ad4a6d80e8b7d248a79bb0477e35ef61f7a2177de5b1d97ed136fb523aa"} Aug 04 16:06:34.206714 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:34.206701 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5" event={"ID":"55f89be8-4514-49a5-9192-33a6d150864d","Type":"ContainerDied","Data":"ad00bd528024d0098ce5eb0eb66cd44eb72c978a18422154a7845e039b3af0b4"} Aug 04 16:06:34.206867 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:34.206722 2573 scope.go:117] "RemoveContainer" containerID="884a0ad4a6d80e8b7d248a79bb0477e35ef61f7a2177de5b1d97ed136fb523aa" Aug 04 16:06:34.214280 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:34.214252 2573 scope.go:117] "RemoveContainer" containerID="884a0ad4a6d80e8b7d248a79bb0477e35ef61f7a2177de5b1d97ed136fb523aa" Aug 04 16:06:34.214562 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:06:34.214531 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"884a0ad4a6d80e8b7d248a79bb0477e35ef61f7a2177de5b1d97ed136fb523aa\": container with ID starting with 884a0ad4a6d80e8b7d248a79bb0477e35ef61f7a2177de5b1d97ed136fb523aa not found: ID does not exist" containerID="884a0ad4a6d80e8b7d248a79bb0477e35ef61f7a2177de5b1d97ed136fb523aa" Aug 04 16:06:34.214622 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:34.214577 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"884a0ad4a6d80e8b7d248a79bb0477e35ef61f7a2177de5b1d97ed136fb523aa"} err="failed to get container status \"884a0ad4a6d80e8b7d248a79bb0477e35ef61f7a2177de5b1d97ed136fb523aa\": rpc error: code = NotFound desc = could not find container \"884a0ad4a6d80e8b7d248a79bb0477e35ef61f7a2177de5b1d97ed136fb523aa\": container with ID starting with 884a0ad4a6d80e8b7d248a79bb0477e35ef61f7a2177de5b1d97ed136fb523aa not found: ID does not exist" Aug 04 16:06:34.303630 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:34.303600 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/55f89be8-4514-49a5-9192-33a6d150864d-istio-csr-dns-cert\") pod \"55f89be8-4514-49a5-9192-33a6d150864d\" (UID: \"55f89be8-4514-49a5-9192-33a6d150864d\") " Aug 04 16:06:34.303630 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:34.303639 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/55f89be8-4514-49a5-9192-33a6d150864d-istio-kubeconfig\") pod \"55f89be8-4514-49a5-9192-33a6d150864d\" (UID: \"55f89be8-4514-49a5-9192-33a6d150864d\") " Aug 04 16:06:34.303870 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:34.303664 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zmrd\" (UniqueName: \"kubernetes.io/projected/55f89be8-4514-49a5-9192-33a6d150864d-kube-api-access-9zmrd\") pod \"55f89be8-4514-49a5-9192-33a6d150864d\" (UID: \"55f89be8-4514-49a5-9192-33a6d150864d\") " Aug 04 16:06:34.303870 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:34.303687 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/55f89be8-4514-49a5-9192-33a6d150864d-istio-csr-ca-configmap\") pod \"55f89be8-4514-49a5-9192-33a6d150864d\" (UID: \"55f89be8-4514-49a5-9192-33a6d150864d\") " Aug 04 16:06:34.303870 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:34.303752 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/55f89be8-4514-49a5-9192-33a6d150864d-cacerts\") pod \"55f89be8-4514-49a5-9192-33a6d150864d\" (UID: \"55f89be8-4514-49a5-9192-33a6d150864d\") " Aug 04 16:06:34.303870 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:34.303823 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f89be8-4514-49a5-9192-33a6d150864d-istio-token\") pod \"55f89be8-4514-49a5-9192-33a6d150864d\" (UID: \"55f89be8-4514-49a5-9192-33a6d150864d\") " Aug 04 16:06:34.303870 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:34.303859 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f89be8-4514-49a5-9192-33a6d150864d-local-certs\") pod \"55f89be8-4514-49a5-9192-33a6d150864d\" (UID: \"55f89be8-4514-49a5-9192-33a6d150864d\") " Aug 04 16:06:34.304427 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:34.304176 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55f89be8-4514-49a5-9192-33a6d150864d-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "55f89be8-4514-49a5-9192-33a6d150864d" (UID: "55f89be8-4514-49a5-9192-33a6d150864d"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:06:34.306277 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:34.306253 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55f89be8-4514-49a5-9192-33a6d150864d-cacerts" (OuterVolumeSpecName: "cacerts") pod "55f89be8-4514-49a5-9192-33a6d150864d" (UID: "55f89be8-4514-49a5-9192-33a6d150864d"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:06:34.306388 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:34.306325 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f89be8-4514-49a5-9192-33a6d150864d-local-certs" (OuterVolumeSpecName: "local-certs") pod "55f89be8-4514-49a5-9192-33a6d150864d" (UID: "55f89be8-4514-49a5-9192-33a6d150864d"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:06:34.306439 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:34.306407 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f89be8-4514-49a5-9192-33a6d150864d-kube-api-access-9zmrd" (OuterVolumeSpecName: "kube-api-access-9zmrd") pod "55f89be8-4514-49a5-9192-33a6d150864d" (UID: "55f89be8-4514-49a5-9192-33a6d150864d"). InnerVolumeSpecName "kube-api-access-9zmrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:06:34.306439 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:34.306395 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55f89be8-4514-49a5-9192-33a6d150864d-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "55f89be8-4514-49a5-9192-33a6d150864d" (UID: "55f89be8-4514-49a5-9192-33a6d150864d"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:06:34.306516 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:34.306465 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f89be8-4514-49a5-9192-33a6d150864d-istio-token" (OuterVolumeSpecName: "istio-token") pod "55f89be8-4514-49a5-9192-33a6d150864d" (UID: "55f89be8-4514-49a5-9192-33a6d150864d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:06:34.306567 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:34.306507 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55f89be8-4514-49a5-9192-33a6d150864d-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "55f89be8-4514-49a5-9192-33a6d150864d" (UID: "55f89be8-4514-49a5-9192-33a6d150864d"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:06:34.404861 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:34.404818 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f89be8-4514-49a5-9192-33a6d150864d-istio-token\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:06:34.404861 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:34.404859 2573 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f89be8-4514-49a5-9192-33a6d150864d-local-certs\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:06:34.405134 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:34.404874 2573 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/55f89be8-4514-49a5-9192-33a6d150864d-istio-csr-dns-cert\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:06:34.405134 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:34.404887 2573 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/55f89be8-4514-49a5-9192-33a6d150864d-istio-kubeconfig\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:06:34.405134 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:34.404901 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zmrd\" (UniqueName: \"kubernetes.io/projected/55f89be8-4514-49a5-9192-33a6d150864d-kube-api-access-9zmrd\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:06:34.405134 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:34.404914 2573 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/55f89be8-4514-49a5-9192-33a6d150864d-istio-csr-ca-configmap\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:06:34.405134 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:34.404928 2573 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/55f89be8-4514-49a5-9192-33a6d150864d-cacerts\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:06:34.529851 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:34.529822 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5"] Aug 04 16:06:34.533687 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:34.533660 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lnsx5"] Aug 04 16:06:35.077380 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:35.077346 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55f89be8-4514-49a5-9192-33a6d150864d" path="/var/lib/kubelet/pods/55f89be8-4514-49a5-9192-33a6d150864d/volumes" Aug 04 16:06:40.965187 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:40.965157 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-gvb4l_c16f82ab-8340-49ee-b41e-5685d4592fd2/console-operator/2.log" Aug 04 16:06:40.965831 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:40.965807 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-gvb4l_c16f82ab-8340-49ee-b41e-5685d4592fd2/console-operator/2.log" Aug 04 16:06:40.969380 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:40.969362 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mtbc5_11f9cbee-a773-420b-9873-5c8cb7679dd8/ovn-acl-logging/0.log" Aug 04 16:06:40.969621 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:40.969605 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mtbc5_11f9cbee-a773-420b-9873-5c8cb7679dd8/ovn-acl-logging/0.log" Aug 04 16:06:44.209165 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:06:44.209132 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5967759b96-fkvhc" Aug 04 16:07:52.397846 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:07:52.397817 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-qkn9q"] Aug 04 16:07:52.398405 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:07:52.398228 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55f89be8-4514-49a5-9192-33a6d150864d" containerName="discovery" Aug 04 16:07:52.398405 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:07:52.398247 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="55f89be8-4514-49a5-9192-33a6d150864d" containerName="discovery" Aug 04 16:07:52.398405 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:07:52.398314 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="55f89be8-4514-49a5-9192-33a6d150864d" containerName="discovery" Aug 04 16:07:52.400398 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:07:52.400380 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-qkn9q" Aug 04 16:07:52.405031 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:07:52.405015 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vk9xc\"" Aug 04 16:07:52.405823 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:07:52.405774 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Aug 04 16:07:52.405823 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:07:52.405808 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Aug 04 16:07:52.406007 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:07:52.405838 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Aug 04 16:07:52.406007 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:07:52.405841 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Aug 04 16:07:52.416922 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:07:52.416883 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-qkn9q"] Aug 04 16:07:52.492585 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:07:52.492538 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p7pw\" (UniqueName: \"kubernetes.io/projected/f411eef7-205b-4846-95e4-fbf086465197-kube-api-access-4p7pw\") pod \"kuadrant-console-plugin-598f86468b-qkn9q\" (UID: \"f411eef7-205b-4846-95e4-fbf086465197\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-qkn9q" Aug 04 16:07:52.492766 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:07:52.492602 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f411eef7-205b-4846-95e4-fbf086465197-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-qkn9q\" (UID: \"f411eef7-205b-4846-95e4-fbf086465197\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-qkn9q" Aug 04 16:07:52.492766 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:07:52.492643 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f411eef7-205b-4846-95e4-fbf086465197-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-qkn9q\" (UID: \"f411eef7-205b-4846-95e4-fbf086465197\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-qkn9q" Aug 04 16:07:52.593145 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:07:52.593111 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f411eef7-205b-4846-95e4-fbf086465197-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-qkn9q\" (UID: \"f411eef7-205b-4846-95e4-fbf086465197\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-qkn9q" Aug 04 16:07:52.593336 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:07:52.593206 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f411eef7-205b-4846-95e4-fbf086465197-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-qkn9q\" (UID: \"f411eef7-205b-4846-95e4-fbf086465197\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-qkn9q" Aug 04 16:07:52.593336 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:07:52.593270 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p7pw\" (UniqueName: \"kubernetes.io/projected/f411eef7-205b-4846-95e4-fbf086465197-kube-api-access-4p7pw\") pod \"kuadrant-console-plugin-598f86468b-qkn9q\" (UID: \"f411eef7-205b-4846-95e4-fbf086465197\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-qkn9q" Aug 04 16:07:52.593452 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:07:52.593335 2573 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Aug 04 16:07:52.593452 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:07:52.593428 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f411eef7-205b-4846-95e4-fbf086465197-plugin-serving-cert podName:f411eef7-205b-4846-95e4-fbf086465197 nodeName:}" failed. No retries permitted until 2026-08-04 16:07:53.093403818 +0000 UTC m=+672.510945938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/f411eef7-205b-4846-95e4-fbf086465197-plugin-serving-cert") pod "kuadrant-console-plugin-598f86468b-qkn9q" (UID: "f411eef7-205b-4846-95e4-fbf086465197") : secret "plugin-serving-cert" not found Aug 04 16:07:52.593880 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:07:52.593861 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f411eef7-205b-4846-95e4-fbf086465197-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-qkn9q\" (UID: \"f411eef7-205b-4846-95e4-fbf086465197\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-qkn9q" Aug 04 16:07:52.602028 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:07:52.602011 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p7pw\" (UniqueName: \"kubernetes.io/projected/f411eef7-205b-4846-95e4-fbf086465197-kube-api-access-4p7pw\") pod \"kuadrant-console-plugin-598f86468b-qkn9q\" (UID: \"f411eef7-205b-4846-95e4-fbf086465197\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-qkn9q" Aug 04 16:07:53.098175 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:07:53.098142 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f411eef7-205b-4846-95e4-fbf086465197-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-qkn9q\" (UID: \"f411eef7-205b-4846-95e4-fbf086465197\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-qkn9q" Aug 04 16:07:53.100515 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:07:53.100483 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f411eef7-205b-4846-95e4-fbf086465197-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-qkn9q\" (UID: \"f411eef7-205b-4846-95e4-fbf086465197\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-qkn9q" Aug 04 16:07:53.309692 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:07:53.309650 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-qkn9q" Aug 04 16:07:53.441672 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:07:53.441643 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-qkn9q"] Aug 04 16:07:53.444952 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:07:53.444926 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf411eef7_205b_4846_95e4_fbf086465197.slice/crio-eb6bb99b5eccc7a7f65ca75d1c238fa2560cf300d444e147df7b55d541ade54b WatchSource:0}: Error finding container eb6bb99b5eccc7a7f65ca75d1c238fa2560cf300d444e147df7b55d541ade54b: Status 404 returned error can't find the container with id eb6bb99b5eccc7a7f65ca75d1c238fa2560cf300d444e147df7b55d541ade54b Aug 04 16:07:53.490734 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:07:53.490702 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-qkn9q" event={"ID":"f411eef7-205b-4846-95e4-fbf086465197","Type":"ContainerStarted","Data":"eb6bb99b5eccc7a7f65ca75d1c238fa2560cf300d444e147df7b55d541ade54b"} Aug 04 16:08:00.521503 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:00.521466 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-qkn9q" event={"ID":"f411eef7-205b-4846-95e4-fbf086465197","Type":"ContainerStarted","Data":"86017fd075333ca83bad265dd369a166d0c3e1e31398439bc8f70170011b89d4"} Aug 04 16:08:00.542290 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:00.542234 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-qkn9q" podStartSLOduration=2.043172534 podStartE2EDuration="8.542216994s" totalImagesPullingTime="6.49904446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:07:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:08:00.541428765 +0000 UTC m=+679.958970893" watchObservedRunningTime="2026-08-04 16:08:00.542216994 +0000 UTC m=+679.959759129" Aug 04 16:08:40.292412 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:40.292331 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-tklt2"] Aug 04 16:08:40.296772 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:40.296744 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-tklt2" Aug 04 16:08:40.299425 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:40.299405 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rv57c\"" Aug 04 16:08:40.307073 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:40.307046 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-tklt2"] Aug 04 16:08:40.368182 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:40.368152 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kvmw\" (UniqueName: \"kubernetes.io/projected/5807adb1-9e19-4a98-ad0e-cb5fc23de321-kube-api-access-7kvmw\") pod \"authorino-b54df94b4-tklt2\" (UID: \"5807adb1-9e19-4a98-ad0e-cb5fc23de321\") " pod="kuadrant-system/authorino-b54df94b4-tklt2" Aug 04 16:08:40.468833 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:40.468793 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kvmw\" (UniqueName: \"kubernetes.io/projected/5807adb1-9e19-4a98-ad0e-cb5fc23de321-kube-api-access-7kvmw\") pod \"authorino-b54df94b4-tklt2\" (UID: \"5807adb1-9e19-4a98-ad0e-cb5fc23de321\") " pod="kuadrant-system/authorino-b54df94b4-tklt2" Aug 04 16:08:40.476971 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:40.476940 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kvmw\" (UniqueName: \"kubernetes.io/projected/5807adb1-9e19-4a98-ad0e-cb5fc23de321-kube-api-access-7kvmw\") pod \"authorino-b54df94b4-tklt2\" (UID: \"5807adb1-9e19-4a98-ad0e-cb5fc23de321\") " pod="kuadrant-system/authorino-b54df94b4-tklt2" Aug 04 16:08:40.496474 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:40.496445 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-8s79d"] Aug 04 16:08:40.499228 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:40.499212 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-8s79d" Aug 04 16:08:40.508005 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:40.507978 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-8s79d"] Aug 04 16:08:40.569577 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:40.569471 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8lwx\" (UniqueName: \"kubernetes.io/projected/12f99c19-ff72-45ea-aa78-65ba1ac4fcbc-kube-api-access-h8lwx\") pod \"authorino-c548b5d7d-8s79d\" (UID: \"12f99c19-ff72-45ea-aa78-65ba1ac4fcbc\") " pod="kuadrant-system/authorino-c548b5d7d-8s79d" Aug 04 16:08:40.606504 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:40.606467 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-tklt2" Aug 04 16:08:40.670649 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:40.670610 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8lwx\" (UniqueName: \"kubernetes.io/projected/12f99c19-ff72-45ea-aa78-65ba1ac4fcbc-kube-api-access-h8lwx\") pod \"authorino-c548b5d7d-8s79d\" (UID: \"12f99c19-ff72-45ea-aa78-65ba1ac4fcbc\") " pod="kuadrant-system/authorino-c548b5d7d-8s79d" Aug 04 16:08:40.686886 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:40.686861 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8lwx\" (UniqueName: \"kubernetes.io/projected/12f99c19-ff72-45ea-aa78-65ba1ac4fcbc-kube-api-access-h8lwx\") pod \"authorino-c548b5d7d-8s79d\" (UID: \"12f99c19-ff72-45ea-aa78-65ba1ac4fcbc\") " pod="kuadrant-system/authorino-c548b5d7d-8s79d" Aug 04 16:08:40.738249 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:40.738184 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-tklt2"] Aug 04 16:08:40.742108 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:08:40.742082 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5807adb1_9e19_4a98_ad0e_cb5fc23de321.slice/crio-5354dacbb24db1e077bd682c17699ee159800c4ae27fce6e2a2385f67d56f963 WatchSource:0}: Error finding container 5354dacbb24db1e077bd682c17699ee159800c4ae27fce6e2a2385f67d56f963: Status 404 returned error can't find the container with id 5354dacbb24db1e077bd682c17699ee159800c4ae27fce6e2a2385f67d56f963 Aug 04 16:08:40.809168 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:40.809129 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-8s79d" Aug 04 16:08:40.939465 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:40.939431 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-8s79d"] Aug 04 16:08:40.942694 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:08:40.942667 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12f99c19_ff72_45ea_aa78_65ba1ac4fcbc.slice/crio-582a3a8bc5cb334ea69a6a7a051e04fb35ccd5eaf8d52d2c8ab3de88a15185b6 WatchSource:0}: Error finding container 582a3a8bc5cb334ea69a6a7a051e04fb35ccd5eaf8d52d2c8ab3de88a15185b6: Status 404 returned error can't find the container with id 582a3a8bc5cb334ea69a6a7a051e04fb35ccd5eaf8d52d2c8ab3de88a15185b6 Aug 04 16:08:41.674859 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:41.674796 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-8s79d" event={"ID":"12f99c19-ff72-45ea-aa78-65ba1ac4fcbc","Type":"ContainerStarted","Data":"582a3a8bc5cb334ea69a6a7a051e04fb35ccd5eaf8d52d2c8ab3de88a15185b6"} Aug 04 16:08:41.676276 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:41.676249 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-tklt2" event={"ID":"5807adb1-9e19-4a98-ad0e-cb5fc23de321","Type":"ContainerStarted","Data":"5354dacbb24db1e077bd682c17699ee159800c4ae27fce6e2a2385f67d56f963"} Aug 04 16:08:43.686261 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:43.686176 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-tklt2" event={"ID":"5807adb1-9e19-4a98-ad0e-cb5fc23de321","Type":"ContainerStarted","Data":"0ac1618fed8c27d3b3a6c20bd5c42f2bc7550707c7d8a67c3814e8f5a3acb4c3"} Aug 04 16:08:46.699708 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:46.699677 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-8s79d" event={"ID":"12f99c19-ff72-45ea-aa78-65ba1ac4fcbc","Type":"ContainerStarted","Data":"0d0934e3f79b914a725419b46b0f515933d717830ac3e3d24d4723509afed5c9"} Aug 04 16:08:46.714841 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:46.714795 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-8s79d" podStartSLOduration=1.350213177 podStartE2EDuration="6.714776817s" totalImagesPullingTime="5.36456364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:08:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:08:46.714704905 +0000 UTC m=+726.132247033" watchObservedRunningTime="2026-08-04 16:08:46.714776817 +0000 UTC m=+726.132318941" Aug 04 16:08:46.715891 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:46.715854 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-tklt2" podStartSLOduration=4.098474334 podStartE2EDuration="6.71583797s" totalImagesPullingTime="2.617363636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:08:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:08:43.711204637 +0000 UTC m=+723.128746764" watchObservedRunningTime="2026-08-04 16:08:46.71583797 +0000 UTC m=+726.133380097" Aug 04 16:08:46.740870 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:46.740838 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-tklt2"] Aug 04 16:08:46.741034 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:46.740995 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-tklt2" podUID="5807adb1-9e19-4a98-ad0e-cb5fc23de321" containerName="authorino" containerID="cri-o://0ac1618fed8c27d3b3a6c20bd5c42f2bc7550707c7d8a67c3814e8f5a3acb4c3" gracePeriod=30 Aug 04 16:08:46.982410 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:46.982385 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-tklt2" Aug 04 16:08:47.021269 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:47.021217 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kvmw\" (UniqueName: \"kubernetes.io/projected/5807adb1-9e19-4a98-ad0e-cb5fc23de321-kube-api-access-7kvmw\") pod \"5807adb1-9e19-4a98-ad0e-cb5fc23de321\" (UID: \"5807adb1-9e19-4a98-ad0e-cb5fc23de321\") " Aug 04 16:08:47.023357 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:47.023331 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5807adb1-9e19-4a98-ad0e-cb5fc23de321-kube-api-access-7kvmw" (OuterVolumeSpecName: "kube-api-access-7kvmw") pod "5807adb1-9e19-4a98-ad0e-cb5fc23de321" (UID: "5807adb1-9e19-4a98-ad0e-cb5fc23de321"). InnerVolumeSpecName "kube-api-access-7kvmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:08:47.121873 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:47.121825 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kvmw\" (UniqueName: \"kubernetes.io/projected/5807adb1-9e19-4a98-ad0e-cb5fc23de321-kube-api-access-7kvmw\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:08:47.706969 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:47.706930 2573 generic.go:356] "Generic (PLEG): container finished" podID="5807adb1-9e19-4a98-ad0e-cb5fc23de321" containerID="0ac1618fed8c27d3b3a6c20bd5c42f2bc7550707c7d8a67c3814e8f5a3acb4c3" exitCode=0 Aug 04 16:08:47.706969 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:47.706974 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-tklt2" Aug 04 16:08:47.707419 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:47.707020 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-tklt2" event={"ID":"5807adb1-9e19-4a98-ad0e-cb5fc23de321","Type":"ContainerDied","Data":"0ac1618fed8c27d3b3a6c20bd5c42f2bc7550707c7d8a67c3814e8f5a3acb4c3"} Aug 04 16:08:47.707419 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:47.707060 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-tklt2" event={"ID":"5807adb1-9e19-4a98-ad0e-cb5fc23de321","Type":"ContainerDied","Data":"5354dacbb24db1e077bd682c17699ee159800c4ae27fce6e2a2385f67d56f963"} Aug 04 16:08:47.707419 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:47.707081 2573 scope.go:117] "RemoveContainer" containerID="0ac1618fed8c27d3b3a6c20bd5c42f2bc7550707c7d8a67c3814e8f5a3acb4c3" Aug 04 16:08:47.714980 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:47.714959 2573 scope.go:117] "RemoveContainer" containerID="0ac1618fed8c27d3b3a6c20bd5c42f2bc7550707c7d8a67c3814e8f5a3acb4c3" Aug 04 16:08:47.715220 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:08:47.715201 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ac1618fed8c27d3b3a6c20bd5c42f2bc7550707c7d8a67c3814e8f5a3acb4c3\": container with ID starting with 0ac1618fed8c27d3b3a6c20bd5c42f2bc7550707c7d8a67c3814e8f5a3acb4c3 not found: ID does not exist" containerID="0ac1618fed8c27d3b3a6c20bd5c42f2bc7550707c7d8a67c3814e8f5a3acb4c3" Aug 04 16:08:47.715282 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:47.715228 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ac1618fed8c27d3b3a6c20bd5c42f2bc7550707c7d8a67c3814e8f5a3acb4c3"} err="failed to get container status \"0ac1618fed8c27d3b3a6c20bd5c42f2bc7550707c7d8a67c3814e8f5a3acb4c3\": rpc error: code = NotFound desc = could not find container \"0ac1618fed8c27d3b3a6c20bd5c42f2bc7550707c7d8a67c3814e8f5a3acb4c3\": container with ID starting with 0ac1618fed8c27d3b3a6c20bd5c42f2bc7550707c7d8a67c3814e8f5a3acb4c3 not found: ID does not exist" Aug 04 16:08:47.721699 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:47.721676 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-tklt2"] Aug 04 16:08:47.726023 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:47.726001 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-tklt2"] Aug 04 16:08:49.077341 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:08:49.077309 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5807adb1-9e19-4a98-ad0e-cb5fc23de321" path="/var/lib/kubelet/pods/5807adb1-9e19-4a98-ad0e-cb5fc23de321/volumes" Aug 04 16:09:08.111446 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:08.111410 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-ls86s"] Aug 04 16:09:08.111862 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:08.111700 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5807adb1-9e19-4a98-ad0e-cb5fc23de321" containerName="authorino" Aug 04 16:09:08.111862 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:08.111711 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5807adb1-9e19-4a98-ad0e-cb5fc23de321" containerName="authorino" Aug 04 16:09:08.111862 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:08.111760 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5807adb1-9e19-4a98-ad0e-cb5fc23de321" containerName="authorino" Aug 04 16:09:08.114667 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:08.114651 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-ls86s" Aug 04 16:09:08.148762 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:08.148729 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-ls86s"] Aug 04 16:09:08.179877 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:08.179847 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdzqm\" (UniqueName: \"kubernetes.io/projected/13ebea02-28ae-4a32-a080-1001741c1aaf-kube-api-access-zdzqm\") pod \"authorino-5dc7ccb5f-ls86s\" (UID: \"13ebea02-28ae-4a32-a080-1001741c1aaf\") " pod="kuadrant-system/authorino-5dc7ccb5f-ls86s" Aug 04 16:09:08.235661 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:08.235624 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-ls86s"] Aug 04 16:09:08.235891 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:09:08.235871 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-zdzqm], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5dc7ccb5f-ls86s" podUID="13ebea02-28ae-4a32-a080-1001741c1aaf" Aug 04 16:09:08.280677 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:08.280641 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdzqm\" (UniqueName: \"kubernetes.io/projected/13ebea02-28ae-4a32-a080-1001741c1aaf-kube-api-access-zdzqm\") pod \"authorino-5dc7ccb5f-ls86s\" (UID: \"13ebea02-28ae-4a32-a080-1001741c1aaf\") " pod="kuadrant-system/authorino-5dc7ccb5f-ls86s" Aug 04 16:09:08.300359 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:08.300324 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdzqm\" (UniqueName: \"kubernetes.io/projected/13ebea02-28ae-4a32-a080-1001741c1aaf-kube-api-access-zdzqm\") pod \"authorino-5dc7ccb5f-ls86s\" (UID: \"13ebea02-28ae-4a32-a080-1001741c1aaf\") " pod="kuadrant-system/authorino-5dc7ccb5f-ls86s" Aug 04 16:09:08.789733 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:08.789700 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-ls86s" Aug 04 16:09:08.796641 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:08.796614 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-ls86s" Aug 04 16:09:08.885799 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:08.885769 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdzqm\" (UniqueName: \"kubernetes.io/projected/13ebea02-28ae-4a32-a080-1001741c1aaf-kube-api-access-zdzqm\") pod \"13ebea02-28ae-4a32-a080-1001741c1aaf\" (UID: \"13ebea02-28ae-4a32-a080-1001741c1aaf\") " Aug 04 16:09:08.887920 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:08.887893 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13ebea02-28ae-4a32-a080-1001741c1aaf-kube-api-access-zdzqm" (OuterVolumeSpecName: "kube-api-access-zdzqm") pod "13ebea02-28ae-4a32-a080-1001741c1aaf" (UID: "13ebea02-28ae-4a32-a080-1001741c1aaf"). InnerVolumeSpecName "kube-api-access-zdzqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:09:08.987106 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:08.987068 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdzqm\" (UniqueName: \"kubernetes.io/projected/13ebea02-28ae-4a32-a080-1001741c1aaf-kube-api-access-zdzqm\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:09:09.793915 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:09.793882 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-ls86s" Aug 04 16:09:09.825015 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:09.824983 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-ls86s"] Aug 04 16:09:09.830980 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:09.830954 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-ls86s"] Aug 04 16:09:11.077712 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:11.077672 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13ebea02-28ae-4a32-a080-1001741c1aaf" path="/var/lib/kubelet/pods/13ebea02-28ae-4a32-a080-1001741c1aaf/volumes" Aug 04 16:09:11.626756 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:11.626723 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-8s79d"] Aug 04 16:09:11.626953 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:11.626907 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-8s79d" podUID="12f99c19-ff72-45ea-aa78-65ba1ac4fcbc" containerName="authorino" containerID="cri-o://0d0934e3f79b914a725419b46b0f515933d717830ac3e3d24d4723509afed5c9" gracePeriod=30 Aug 04 16:09:11.804966 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:11.804932 2573 generic.go:356] "Generic (PLEG): container finished" podID="12f99c19-ff72-45ea-aa78-65ba1ac4fcbc" containerID="0d0934e3f79b914a725419b46b0f515933d717830ac3e3d24d4723509afed5c9" exitCode=0 Aug 04 16:09:11.805107 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:11.805006 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-8s79d" event={"ID":"12f99c19-ff72-45ea-aa78-65ba1ac4fcbc","Type":"ContainerDied","Data":"0d0934e3f79b914a725419b46b0f515933d717830ac3e3d24d4723509afed5c9"} Aug 04 16:09:11.860109 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:11.860083 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-8s79d" Aug 04 16:09:11.911492 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:11.911401 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8lwx\" (UniqueName: \"kubernetes.io/projected/12f99c19-ff72-45ea-aa78-65ba1ac4fcbc-kube-api-access-h8lwx\") pod \"12f99c19-ff72-45ea-aa78-65ba1ac4fcbc\" (UID: \"12f99c19-ff72-45ea-aa78-65ba1ac4fcbc\") " Aug 04 16:09:11.913446 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:11.913412 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12f99c19-ff72-45ea-aa78-65ba1ac4fcbc-kube-api-access-h8lwx" (OuterVolumeSpecName: "kube-api-access-h8lwx") pod "12f99c19-ff72-45ea-aa78-65ba1ac4fcbc" (UID: "12f99c19-ff72-45ea-aa78-65ba1ac4fcbc"). InnerVolumeSpecName "kube-api-access-h8lwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:09:12.012744 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:12.012703 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8lwx\" (UniqueName: \"kubernetes.io/projected/12f99c19-ff72-45ea-aa78-65ba1ac4fcbc-kube-api-access-h8lwx\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:09:12.809858 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:12.809828 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-8s79d" Aug 04 16:09:12.810299 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:12.809833 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-8s79d" event={"ID":"12f99c19-ff72-45ea-aa78-65ba1ac4fcbc","Type":"ContainerDied","Data":"582a3a8bc5cb334ea69a6a7a051e04fb35ccd5eaf8d52d2c8ab3de88a15185b6"} Aug 04 16:09:12.810299 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:12.809933 2573 scope.go:117] "RemoveContainer" containerID="0d0934e3f79b914a725419b46b0f515933d717830ac3e3d24d4723509afed5c9" Aug 04 16:09:12.829974 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:12.829950 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-8s79d"] Aug 04 16:09:12.834373 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:12.834351 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-8s79d"] Aug 04 16:09:13.077994 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:13.077917 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12f99c19-ff72-45ea-aa78-65ba1ac4fcbc" path="/var/lib/kubelet/pods/12f99c19-ff72-45ea-aa78-65ba1ac4fcbc/volumes" Aug 04 16:09:16.724388 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:16.724351 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d474f94c9-kgcpp"] Aug 04 16:09:16.724808 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:16.724659 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12f99c19-ff72-45ea-aa78-65ba1ac4fcbc" containerName="authorino" Aug 04 16:09:16.724808 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:16.724671 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="12f99c19-ff72-45ea-aa78-65ba1ac4fcbc" containerName="authorino" Aug 04 16:09:16.724808 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:16.724727 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="12f99c19-ff72-45ea-aa78-65ba1ac4fcbc" containerName="authorino" Aug 04 16:09:16.732131 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:16.732110 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d474f94c9-kgcpp" Aug 04 16:09:16.735044 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:16.735020 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Aug 04 16:09:16.735155 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:16.735025 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Aug 04 16:09:16.735155 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:16.735070 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-42s8l\"" Aug 04 16:09:16.737752 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:16.737727 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d474f94c9-kgcpp"] Aug 04 16:09:16.852829 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:16.852795 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/83bfb81e-bf2d-468c-948a-3e5b25829dfb-webhook-cert\") pod \"maas-controller-d474f94c9-kgcpp\" (UID: \"83bfb81e-bf2d-468c-948a-3e5b25829dfb\") " pod="opendatahub/maas-controller-d474f94c9-kgcpp" Aug 04 16:09:16.852991 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:16.852840 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnnwr\" (UniqueName: \"kubernetes.io/projected/83bfb81e-bf2d-468c-948a-3e5b25829dfb-kube-api-access-fnnwr\") pod \"maas-controller-d474f94c9-kgcpp\" (UID: \"83bfb81e-bf2d-468c-948a-3e5b25829dfb\") " pod="opendatahub/maas-controller-d474f94c9-kgcpp" Aug 04 16:09:16.953386 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:16.953345 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/83bfb81e-bf2d-468c-948a-3e5b25829dfb-webhook-cert\") pod \"maas-controller-d474f94c9-kgcpp\" (UID: \"83bfb81e-bf2d-468c-948a-3e5b25829dfb\") " pod="opendatahub/maas-controller-d474f94c9-kgcpp" Aug 04 16:09:16.953626 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:16.953401 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnnwr\" (UniqueName: \"kubernetes.io/projected/83bfb81e-bf2d-468c-948a-3e5b25829dfb-kube-api-access-fnnwr\") pod \"maas-controller-d474f94c9-kgcpp\" (UID: \"83bfb81e-bf2d-468c-948a-3e5b25829dfb\") " pod="opendatahub/maas-controller-d474f94c9-kgcpp" Aug 04 16:09:16.955905 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:16.955885 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/83bfb81e-bf2d-468c-948a-3e5b25829dfb-webhook-cert\") pod \"maas-controller-d474f94c9-kgcpp\" (UID: \"83bfb81e-bf2d-468c-948a-3e5b25829dfb\") " pod="opendatahub/maas-controller-d474f94c9-kgcpp" Aug 04 16:09:16.962682 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:16.962660 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnnwr\" (UniqueName: \"kubernetes.io/projected/83bfb81e-bf2d-468c-948a-3e5b25829dfb-kube-api-access-fnnwr\") pod \"maas-controller-d474f94c9-kgcpp\" (UID: \"83bfb81e-bf2d-468c-948a-3e5b25829dfb\") " pod="opendatahub/maas-controller-d474f94c9-kgcpp" Aug 04 16:09:17.044376 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:17.044299 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d474f94c9-kgcpp" Aug 04 16:09:17.215493 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:17.215461 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d474f94c9-kgcpp"] Aug 04 16:09:17.218741 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:09:17.218710 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83bfb81e_bf2d_468c_948a_3e5b25829dfb.slice/crio-edea0f01b921632caf3f80834cc23798c6b85dbb0f0f9b15fe3239e0e9cc9efb WatchSource:0}: Error finding container edea0f01b921632caf3f80834cc23798c6b85dbb0f0f9b15fe3239e0e9cc9efb: Status 404 returned error can't find the container with id edea0f01b921632caf3f80834cc23798c6b85dbb0f0f9b15fe3239e0e9cc9efb Aug 04 16:09:17.830226 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:17.830191 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d474f94c9-kgcpp" event={"ID":"83bfb81e-bf2d-468c-948a-3e5b25829dfb","Type":"ContainerStarted","Data":"edea0f01b921632caf3f80834cc23798c6b85dbb0f0f9b15fe3239e0e9cc9efb"} Aug 04 16:09:19.840794 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:19.840758 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d474f94c9-kgcpp" event={"ID":"83bfb81e-bf2d-468c-948a-3e5b25829dfb","Type":"ContainerStarted","Data":"f0ef97effe939cbeecd5de39c7e9bc2a42c33aa902137d0daa27cb1e11f0ef1e"} Aug 04 16:09:19.841236 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:19.840855 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d474f94c9-kgcpp" Aug 04 16:09:19.877871 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:19.877817 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-d474f94c9-kgcpp" podStartSLOduration=1.381236318 podStartE2EDuration="3.877796124s" totalImagesPullingTime="2.496559806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:09:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:09:19.876615572 +0000 UTC m=+759.294157783" watchObservedRunningTime="2026-08-04 16:09:19.877796124 +0000 UTC m=+759.295338249" Aug 04 16:09:30.847577 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:30.847538 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d474f94c9-kgcpp" Aug 04 16:09:54.666481 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:54.666450 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-699c89f89b-794wr"] Aug 04 16:09:54.674153 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:54.674131 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-699c89f89b-794wr" Aug 04 16:09:54.680573 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:54.677627 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 16:09:54.680573 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:54.677959 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q4sc5\"" Aug 04 16:09:54.680573 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:54.678323 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 04 16:09:54.680573 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:54.678750 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Aug 04 16:09:54.683033 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:54.683007 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-699c89f89b-794wr"] Aug 04 16:09:54.747424 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:54.747384 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b3931da-3b90-4dbe-9651-9aa3cf4b6985-maas-api-tls\") pod \"maas-api-699c89f89b-794wr\" (UID: \"6b3931da-3b90-4dbe-9651-9aa3cf4b6985\") " pod="odh-ai-gateway-infra/maas-api-699c89f89b-794wr" Aug 04 16:09:54.747623 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:54.747465 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3931da-3b90-4dbe-9651-9aa3cf4b6985-openshift-service-ca\") pod \"maas-api-699c89f89b-794wr\" (UID: \"6b3931da-3b90-4dbe-9651-9aa3cf4b6985\") " pod="odh-ai-gateway-infra/maas-api-699c89f89b-794wr" Aug 04 16:09:54.747623 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:54.747489 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-256k7\" (UniqueName: \"kubernetes.io/projected/6b3931da-3b90-4dbe-9651-9aa3cf4b6985-kube-api-access-256k7\") pod \"maas-api-699c89f89b-794wr\" (UID: \"6b3931da-3b90-4dbe-9651-9aa3cf4b6985\") " pod="odh-ai-gateway-infra/maas-api-699c89f89b-794wr" Aug 04 16:09:54.848531 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:54.848495 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3931da-3b90-4dbe-9651-9aa3cf4b6985-openshift-service-ca\") pod \"maas-api-699c89f89b-794wr\" (UID: \"6b3931da-3b90-4dbe-9651-9aa3cf4b6985\") " pod="odh-ai-gateway-infra/maas-api-699c89f89b-794wr" Aug 04 16:09:54.848531 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:54.848533 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-256k7\" (UniqueName: \"kubernetes.io/projected/6b3931da-3b90-4dbe-9651-9aa3cf4b6985-kube-api-access-256k7\") pod \"maas-api-699c89f89b-794wr\" (UID: \"6b3931da-3b90-4dbe-9651-9aa3cf4b6985\") " pod="odh-ai-gateway-infra/maas-api-699c89f89b-794wr" Aug 04 16:09:54.848809 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:54.848596 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b3931da-3b90-4dbe-9651-9aa3cf4b6985-maas-api-tls\") pod \"maas-api-699c89f89b-794wr\" (UID: \"6b3931da-3b90-4dbe-9651-9aa3cf4b6985\") " pod="odh-ai-gateway-infra/maas-api-699c89f89b-794wr" Aug 04 16:09:54.849140 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:54.849118 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3931da-3b90-4dbe-9651-9aa3cf4b6985-openshift-service-ca\") pod \"maas-api-699c89f89b-794wr\" (UID: \"6b3931da-3b90-4dbe-9651-9aa3cf4b6985\") " pod="odh-ai-gateway-infra/maas-api-699c89f89b-794wr" Aug 04 16:09:54.850989 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:54.850972 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b3931da-3b90-4dbe-9651-9aa3cf4b6985-maas-api-tls\") pod \"maas-api-699c89f89b-794wr\" (UID: \"6b3931da-3b90-4dbe-9651-9aa3cf4b6985\") " pod="odh-ai-gateway-infra/maas-api-699c89f89b-794wr" Aug 04 16:09:54.858529 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:54.858500 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-256k7\" (UniqueName: \"kubernetes.io/projected/6b3931da-3b90-4dbe-9651-9aa3cf4b6985-kube-api-access-256k7\") pod \"maas-api-699c89f89b-794wr\" (UID: \"6b3931da-3b90-4dbe-9651-9aa3cf4b6985\") " pod="odh-ai-gateway-infra/maas-api-699c89f89b-794wr" Aug 04 16:09:54.991227 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:54.991196 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-699c89f89b-794wr" Aug 04 16:09:55.129449 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:55.129412 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-699c89f89b-794wr"] Aug 04 16:09:55.133057 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:09:55.133029 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b3931da_3b90_4dbe_9651_9aa3cf4b6985.slice/crio-e910b3b0fe49a7fea99f1158c3a2f92d8c20505bda67d6ed0f9c7b77437928b6 WatchSource:0}: Error finding container e910b3b0fe49a7fea99f1158c3a2f92d8c20505bda67d6ed0f9c7b77437928b6: Status 404 returned error can't find the container with id e910b3b0fe49a7fea99f1158c3a2f92d8c20505bda67d6ed0f9c7b77437928b6 Aug 04 16:09:55.336833 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:55.336756 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-8nbg7"] Aug 04 16:09:55.341629 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:55.341611 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-8nbg7" Aug 04 16:09:55.344965 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:55.344939 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Aug 04 16:09:55.345106 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:55.344991 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-r2mvf\"" Aug 04 16:09:55.353085 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:55.353065 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af2ed0f3-c0b7-4dcd-83bd-90df1ca10aa4-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-8nbg7\" (UID: \"af2ed0f3-c0b7-4dcd-83bd-90df1ca10aa4\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-8nbg7" Aug 04 16:09:55.353183 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:55.353101 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7drjk\" (UniqueName: \"kubernetes.io/projected/af2ed0f3-c0b7-4dcd-83bd-90df1ca10aa4-kube-api-access-7drjk\") pod \"payload-pre-processing-687dbb6cf6-8nbg7\" (UID: \"af2ed0f3-c0b7-4dcd-83bd-90df1ca10aa4\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-8nbg7" Aug 04 16:09:55.367874 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:55.367845 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-8nbg7"] Aug 04 16:09:55.453777 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:55.453732 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af2ed0f3-c0b7-4dcd-83bd-90df1ca10aa4-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-8nbg7\" (UID: \"af2ed0f3-c0b7-4dcd-83bd-90df1ca10aa4\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-8nbg7" Aug 04 16:09:55.453971 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:55.453799 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7drjk\" (UniqueName: \"kubernetes.io/projected/af2ed0f3-c0b7-4dcd-83bd-90df1ca10aa4-kube-api-access-7drjk\") pod \"payload-pre-processing-687dbb6cf6-8nbg7\" (UID: \"af2ed0f3-c0b7-4dcd-83bd-90df1ca10aa4\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-8nbg7" Aug 04 16:09:55.454491 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:55.454472 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af2ed0f3-c0b7-4dcd-83bd-90df1ca10aa4-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-8nbg7\" (UID: \"af2ed0f3-c0b7-4dcd-83bd-90df1ca10aa4\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-8nbg7" Aug 04 16:09:55.469667 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:55.469640 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7drjk\" (UniqueName: \"kubernetes.io/projected/af2ed0f3-c0b7-4dcd-83bd-90df1ca10aa4-kube-api-access-7drjk\") pod \"payload-pre-processing-687dbb6cf6-8nbg7\" (UID: \"af2ed0f3-c0b7-4dcd-83bd-90df1ca10aa4\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-8nbg7" Aug 04 16:09:55.652136 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:55.652051 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-8nbg7" Aug 04 16:09:55.795788 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:55.795756 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-8nbg7"] Aug 04 16:09:55.798881 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:09:55.798852 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf2ed0f3_c0b7_4dcd_83bd_90df1ca10aa4.slice/crio-9f58ae744f7abb8cc5e8a474fa0e18f8b096853fc2469a75ca28780155e1b12b WatchSource:0}: Error finding container 9f58ae744f7abb8cc5e8a474fa0e18f8b096853fc2469a75ca28780155e1b12b: Status 404 returned error can't find the container with id 9f58ae744f7abb8cc5e8a474fa0e18f8b096853fc2469a75ca28780155e1b12b Aug 04 16:09:55.986530 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:55.986494 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-699c89f89b-794wr" event={"ID":"6b3931da-3b90-4dbe-9651-9aa3cf4b6985","Type":"ContainerStarted","Data":"e910b3b0fe49a7fea99f1158c3a2f92d8c20505bda67d6ed0f9c7b77437928b6"} Aug 04 16:09:55.987582 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:55.987541 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-8nbg7" event={"ID":"af2ed0f3-c0b7-4dcd-83bd-90df1ca10aa4","Type":"ContainerStarted","Data":"9f58ae744f7abb8cc5e8a474fa0e18f8b096853fc2469a75ca28780155e1b12b"} Aug 04 16:09:57.998710 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:57.998678 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-699c89f89b-794wr" event={"ID":"6b3931da-3b90-4dbe-9651-9aa3cf4b6985","Type":"ContainerStarted","Data":"2ca7f314f98527ae2b5437123954361635c96b4823685943083ddaf792173baa"} Aug 04 16:09:57.999127 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:57.998779 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-699c89f89b-794wr" Aug 04 16:09:58.021120 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:09:58.021069 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-699c89f89b-794wr" podStartSLOduration=1.59898835 podStartE2EDuration="4.021050616s" totalImagesPullingTime="2.422062266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:09:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:09:58.020295862 +0000 UTC m=+797.437837988" watchObservedRunningTime="2026-08-04 16:09:58.021050616 +0000 UTC m=+797.438592742" Aug 04 16:10:00.009936 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:00.009899 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-8nbg7" event={"ID":"af2ed0f3-c0b7-4dcd-83bd-90df1ca10aa4","Type":"ContainerStarted","Data":"b91337a50979fb07420215ff2b01da364df917055e755546aaa6cddca307e899"} Aug 04 16:10:00.010341 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:00.010033 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-8nbg7" Aug 04 16:10:00.030073 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:00.029757 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-8nbg7" podStartSLOduration=1.205690314 podStartE2EDuration="5.029737519s" totalImagesPullingTime="3.824047205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:09:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:10:00.028343876 +0000 UTC m=+799.445886004" watchObservedRunningTime="2026-08-04 16:10:00.029737519 +0000 UTC m=+799.447279645" Aug 04 16:10:04.011127 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:04.011051 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-699c89f89b-794wr" Aug 04 16:10:11.016965 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:11.016932 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-8nbg7" Aug 04 16:10:34.997467 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:34.997437 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-49b6x"] Aug 04 16:10:35.002016 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:35.001994 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-49b6x" Aug 04 16:10:35.004865 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:35.004838 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-zcdht\"" Aug 04 16:10:35.005014 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:35.004882 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Aug 04 16:10:35.005014 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:35.004838 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Aug 04 16:10:35.005014 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:35.004950 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Aug 04 16:10:35.013716 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:35.013695 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-49b6x"] Aug 04 16:10:35.178473 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:35.178439 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a06359f6-702e-4f7f-974a-2a202ec66ecd-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-49b6x\" (UID: \"a06359f6-702e-4f7f-974a-2a202ec66ecd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-49b6x" Aug 04 16:10:35.178473 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:35.178478 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a06359f6-702e-4f7f-974a-2a202ec66ecd-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-49b6x\" (UID: \"a06359f6-702e-4f7f-974a-2a202ec66ecd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-49b6x" Aug 04 16:10:35.178733 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:35.178501 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a06359f6-702e-4f7f-974a-2a202ec66ecd-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-49b6x\" (UID: \"a06359f6-702e-4f7f-974a-2a202ec66ecd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-49b6x" Aug 04 16:10:35.178733 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:35.178594 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a06359f6-702e-4f7f-974a-2a202ec66ecd-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-49b6x\" (UID: \"a06359f6-702e-4f7f-974a-2a202ec66ecd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-49b6x" Aug 04 16:10:35.178733 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:35.178647 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lswkr\" (UniqueName: \"kubernetes.io/projected/a06359f6-702e-4f7f-974a-2a202ec66ecd-kube-api-access-lswkr\") pod \"e2e-trlp-test-simulated-kserve-598466548d-49b6x\" (UID: \"a06359f6-702e-4f7f-974a-2a202ec66ecd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-49b6x" Aug 04 16:10:35.178733 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:35.178721 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a06359f6-702e-4f7f-974a-2a202ec66ecd-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-49b6x\" (UID: \"a06359f6-702e-4f7f-974a-2a202ec66ecd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-49b6x" Aug 04 16:10:35.279926 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:35.279829 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a06359f6-702e-4f7f-974a-2a202ec66ecd-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-49b6x\" (UID: \"a06359f6-702e-4f7f-974a-2a202ec66ecd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-49b6x" Aug 04 16:10:35.279926 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:35.279874 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a06359f6-702e-4f7f-974a-2a202ec66ecd-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-49b6x\" (UID: \"a06359f6-702e-4f7f-974a-2a202ec66ecd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-49b6x" Aug 04 16:10:35.280143 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:35.279993 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a06359f6-702e-4f7f-974a-2a202ec66ecd-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-49b6x\" (UID: \"a06359f6-702e-4f7f-974a-2a202ec66ecd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-49b6x" Aug 04 16:10:35.280143 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:35.280034 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a06359f6-702e-4f7f-974a-2a202ec66ecd-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-49b6x\" (UID: \"a06359f6-702e-4f7f-974a-2a202ec66ecd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-49b6x" Aug 04 16:10:35.280143 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:35.280068 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a06359f6-702e-4f7f-974a-2a202ec66ecd-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-49b6x\" (UID: \"a06359f6-702e-4f7f-974a-2a202ec66ecd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-49b6x" Aug 04 16:10:35.280143 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:35.280089 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lswkr\" (UniqueName: \"kubernetes.io/projected/a06359f6-702e-4f7f-974a-2a202ec66ecd-kube-api-access-lswkr\") pod \"e2e-trlp-test-simulated-kserve-598466548d-49b6x\" (UID: \"a06359f6-702e-4f7f-974a-2a202ec66ecd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-49b6x" Aug 04 16:10:35.280347 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:35.280270 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a06359f6-702e-4f7f-974a-2a202ec66ecd-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-49b6x\" (UID: \"a06359f6-702e-4f7f-974a-2a202ec66ecd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-49b6x" Aug 04 16:10:35.280431 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:35.280412 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a06359f6-702e-4f7f-974a-2a202ec66ecd-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-49b6x\" (UID: \"a06359f6-702e-4f7f-974a-2a202ec66ecd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-49b6x" Aug 04 16:10:35.280521 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:35.280476 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a06359f6-702e-4f7f-974a-2a202ec66ecd-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-49b6x\" (UID: \"a06359f6-702e-4f7f-974a-2a202ec66ecd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-49b6x" Aug 04 16:10:35.282217 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:35.282196 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a06359f6-702e-4f7f-974a-2a202ec66ecd-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-49b6x\" (UID: \"a06359f6-702e-4f7f-974a-2a202ec66ecd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-49b6x" Aug 04 16:10:35.282397 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:35.282379 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a06359f6-702e-4f7f-974a-2a202ec66ecd-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-49b6x\" (UID: \"a06359f6-702e-4f7f-974a-2a202ec66ecd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-49b6x" Aug 04 16:10:35.289156 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:35.289134 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lswkr\" (UniqueName: \"kubernetes.io/projected/a06359f6-702e-4f7f-974a-2a202ec66ecd-kube-api-access-lswkr\") pod \"e2e-trlp-test-simulated-kserve-598466548d-49b6x\" (UID: \"a06359f6-702e-4f7f-974a-2a202ec66ecd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-49b6x" Aug 04 16:10:35.313098 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:35.313074 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-49b6x" Aug 04 16:10:35.455398 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:35.455365 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-49b6x"] Aug 04 16:10:35.458477 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:10:35.458437 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda06359f6_702e_4f7f_974a_2a202ec66ecd.slice/crio-7d24033230ae71836cb088d9213bbdbeff092359265857501cfe628ee49e540a WatchSource:0}: Error finding container 7d24033230ae71836cb088d9213bbdbeff092359265857501cfe628ee49e540a: Status 404 returned error can't find the container with id 7d24033230ae71836cb088d9213bbdbeff092359265857501cfe628ee49e540a Aug 04 16:10:35.459848 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:35.459829 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 16:10:36.157870 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:36.157831 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-49b6x" event={"ID":"a06359f6-702e-4f7f-974a-2a202ec66ecd","Type":"ContainerStarted","Data":"7d24033230ae71836cb088d9213bbdbeff092359265857501cfe628ee49e540a"} Aug 04 16:10:37.163243 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:37.163205 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-49b6x" event={"ID":"a06359f6-702e-4f7f-974a-2a202ec66ecd","Type":"ContainerStarted","Data":"e2caba12ca29e0a1370e645cec75236db5c2be01c6fb5db120eccc5e3f7af9a8"} Aug 04 16:10:37.184718 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:37.184665 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-49b6x" podStartSLOduration=2.013889581 podStartE2EDuration="3.184645322s" totalImagesPullingTime="1.170755741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:10:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:10:37.18324618 +0000 UTC m=+836.600788426" watchObservedRunningTime="2026-08-04 16:10:37.184645322 +0000 UTC m=+836.602187451" Aug 04 16:10:43.482846 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:43.482809 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79"] Aug 04 16:10:43.486349 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:43.486329 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79" Aug 04 16:10:43.488818 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:43.488796 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Aug 04 16:10:43.496745 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:43.496721 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79"] Aug 04 16:10:43.649494 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:43.649449 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60ee5e6a-8f61-4af0-8b2e-5fa6094a694e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79\" (UID: \"60ee5e6a-8f61-4af0-8b2e-5fa6094a694e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79" Aug 04 16:10:43.649494 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:43.649496 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60ee5e6a-8f61-4af0-8b2e-5fa6094a694e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79\" (UID: \"60ee5e6a-8f61-4af0-8b2e-5fa6094a694e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79" Aug 04 16:10:43.649796 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:43.649514 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60ee5e6a-8f61-4af0-8b2e-5fa6094a694e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79\" (UID: \"60ee5e6a-8f61-4af0-8b2e-5fa6094a694e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79" Aug 04 16:10:43.649796 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:43.649645 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60ee5e6a-8f61-4af0-8b2e-5fa6094a694e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79\" (UID: \"60ee5e6a-8f61-4af0-8b2e-5fa6094a694e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79" Aug 04 16:10:43.649796 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:43.649706 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60ee5e6a-8f61-4af0-8b2e-5fa6094a694e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79\" (UID: \"60ee5e6a-8f61-4af0-8b2e-5fa6094a694e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79" Aug 04 16:10:43.649796 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:43.649731 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5qxm\" (UniqueName: \"kubernetes.io/projected/60ee5e6a-8f61-4af0-8b2e-5fa6094a694e-kube-api-access-x5qxm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79\" (UID: \"60ee5e6a-8f61-4af0-8b2e-5fa6094a694e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79" Aug 04 16:10:43.750488 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:43.750395 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60ee5e6a-8f61-4af0-8b2e-5fa6094a694e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79\" (UID: \"60ee5e6a-8f61-4af0-8b2e-5fa6094a694e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79" Aug 04 16:10:43.750488 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:43.750436 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5qxm\" (UniqueName: \"kubernetes.io/projected/60ee5e6a-8f61-4af0-8b2e-5fa6094a694e-kube-api-access-x5qxm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79\" (UID: \"60ee5e6a-8f61-4af0-8b2e-5fa6094a694e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79" Aug 04 16:10:43.750748 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:43.750507 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60ee5e6a-8f61-4af0-8b2e-5fa6094a694e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79\" (UID: \"60ee5e6a-8f61-4af0-8b2e-5fa6094a694e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79" Aug 04 16:10:43.750748 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:43.750542 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60ee5e6a-8f61-4af0-8b2e-5fa6094a694e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79\" (UID: \"60ee5e6a-8f61-4af0-8b2e-5fa6094a694e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79" Aug 04 16:10:43.750748 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:43.750594 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60ee5e6a-8f61-4af0-8b2e-5fa6094a694e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79\" (UID: \"60ee5e6a-8f61-4af0-8b2e-5fa6094a694e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79" Aug 04 16:10:43.750748 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:43.750632 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60ee5e6a-8f61-4af0-8b2e-5fa6094a694e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79\" (UID: \"60ee5e6a-8f61-4af0-8b2e-5fa6094a694e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79" Aug 04 16:10:43.750950 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:43.750927 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60ee5e6a-8f61-4af0-8b2e-5fa6094a694e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79\" (UID: \"60ee5e6a-8f61-4af0-8b2e-5fa6094a694e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79" Aug 04 16:10:43.750991 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:43.750962 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60ee5e6a-8f61-4af0-8b2e-5fa6094a694e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79\" (UID: \"60ee5e6a-8f61-4af0-8b2e-5fa6094a694e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79" Aug 04 16:10:43.751025 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:43.750983 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60ee5e6a-8f61-4af0-8b2e-5fa6094a694e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79\" (UID: \"60ee5e6a-8f61-4af0-8b2e-5fa6094a694e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79" Aug 04 16:10:43.752827 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:43.752810 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60ee5e6a-8f61-4af0-8b2e-5fa6094a694e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79\" (UID: \"60ee5e6a-8f61-4af0-8b2e-5fa6094a694e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79" Aug 04 16:10:43.753160 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:43.753141 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60ee5e6a-8f61-4af0-8b2e-5fa6094a694e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79\" (UID: \"60ee5e6a-8f61-4af0-8b2e-5fa6094a694e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79" Aug 04 16:10:43.758544 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:43.758526 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5qxm\" (UniqueName: \"kubernetes.io/projected/60ee5e6a-8f61-4af0-8b2e-5fa6094a694e-kube-api-access-x5qxm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79\" (UID: \"60ee5e6a-8f61-4af0-8b2e-5fa6094a694e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79" Aug 04 16:10:43.796459 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:43.796421 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79" Aug 04 16:10:43.925578 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:43.925520 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79"] Aug 04 16:10:43.928829 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:10:43.928806 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60ee5e6a_8f61_4af0_8b2e_5fa6094a694e.slice/crio-2c0dcff3e4afb842b5832d4d9f43ab3f5cfa82ba97b6901b6e3377c100ae3a6b WatchSource:0}: Error finding container 2c0dcff3e4afb842b5832d4d9f43ab3f5cfa82ba97b6901b6e3377c100ae3a6b: Status 404 returned error can't find the container with id 2c0dcff3e4afb842b5832d4d9f43ab3f5cfa82ba97b6901b6e3377c100ae3a6b Aug 04 16:10:44.193062 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:44.193033 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79" event={"ID":"60ee5e6a-8f61-4af0-8b2e-5fa6094a694e","Type":"ContainerStarted","Data":"2c0dcff3e4afb842b5832d4d9f43ab3f5cfa82ba97b6901b6e3377c100ae3a6b"} Aug 04 16:10:45.198840 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:45.198803 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79" event={"ID":"60ee5e6a-8f61-4af0-8b2e-5fa6094a694e","Type":"ContainerStarted","Data":"66a89ec1de7c01326007e79d394e8dd68578641ff5cd2f713e82d7ec6d066e8a"} Aug 04 16:10:45.219121 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:45.219071 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79" podStartSLOduration=1.982305143 podStartE2EDuration="2.219056376s" totalImagesPullingTime="236.751233ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:10:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:10:45.217653421 +0000 UTC m=+844.635195540" watchObservedRunningTime="2026-08-04 16:10:45.219056376 +0000 UTC m=+844.636598530" Aug 04 16:10:45.314283 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:45.314242 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-49b6x" Aug 04 16:10:45.314463 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:45.314312 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-49b6x" Aug 04 16:10:45.326808 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:45.326791 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-49b6x" Aug 04 16:10:46.213868 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:46.213840 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-49b6x" Aug 04 16:10:53.797386 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:53.797348 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79" Aug 04 16:10:53.797825 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:53.797414 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79" Aug 04 16:10:53.809765 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:53.809741 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79" Aug 04 16:10:54.254953 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:10:54.254926 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79" Aug 04 16:11:13.183098 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:11:13.183066 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-m757h"] Aug 04 16:11:13.186907 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:11:13.186872 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m757h" Aug 04 16:11:13.189499 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:11:13.189477 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Aug 04 16:11:13.197208 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:11:13.197187 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-m757h"] Aug 04 16:11:13.286126 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:11:13.286092 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/183a9d7f-d4be-4e9d-9cbd-d51762acf1c9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m757h\" (UID: \"183a9d7f-d4be-4e9d-9cbd-d51762acf1c9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m757h" Aug 04 16:11:13.286126 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:11:13.286128 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/183a9d7f-d4be-4e9d-9cbd-d51762acf1c9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m757h\" (UID: \"183a9d7f-d4be-4e9d-9cbd-d51762acf1c9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m757h" Aug 04 16:11:13.286385 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:11:13.286157 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq24q\" (UniqueName: \"kubernetes.io/projected/183a9d7f-d4be-4e9d-9cbd-d51762acf1c9-kube-api-access-pq24q\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m757h\" (UID: \"183a9d7f-d4be-4e9d-9cbd-d51762acf1c9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m757h" Aug 04 16:11:13.286385 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:11:13.286282 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/183a9d7f-d4be-4e9d-9cbd-d51762acf1c9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m757h\" (UID: \"183a9d7f-d4be-4e9d-9cbd-d51762acf1c9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m757h" Aug 04 16:11:13.286385 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:11:13.286328 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/183a9d7f-d4be-4e9d-9cbd-d51762acf1c9-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m757h\" (UID: \"183a9d7f-d4be-4e9d-9cbd-d51762acf1c9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m757h" Aug 04 16:11:13.286385 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:11:13.286363 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/183a9d7f-d4be-4e9d-9cbd-d51762acf1c9-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m757h\" (UID: \"183a9d7f-d4be-4e9d-9cbd-d51762acf1c9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m757h" Aug 04 16:11:13.387360 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:11:13.387323 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/183a9d7f-d4be-4e9d-9cbd-d51762acf1c9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m757h\" (UID: \"183a9d7f-d4be-4e9d-9cbd-d51762acf1c9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m757h" Aug 04 16:11:13.387360 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:11:13.387369 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/183a9d7f-d4be-4e9d-9cbd-d51762acf1c9-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m757h\" (UID: \"183a9d7f-d4be-4e9d-9cbd-d51762acf1c9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m757h" Aug 04 16:11:13.387648 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:11:13.387391 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/183a9d7f-d4be-4e9d-9cbd-d51762acf1c9-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m757h\" (UID: \"183a9d7f-d4be-4e9d-9cbd-d51762acf1c9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m757h" Aug 04 16:11:13.387648 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:11:13.387530 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/183a9d7f-d4be-4e9d-9cbd-d51762acf1c9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m757h\" (UID: \"183a9d7f-d4be-4e9d-9cbd-d51762acf1c9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m757h" Aug 04 16:11:13.387648 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:11:13.387585 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/183a9d7f-d4be-4e9d-9cbd-d51762acf1c9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m757h\" (UID: \"183a9d7f-d4be-4e9d-9cbd-d51762acf1c9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m757h" Aug 04 16:11:13.387648 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:11:13.387641 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq24q\" (UniqueName: \"kubernetes.io/projected/183a9d7f-d4be-4e9d-9cbd-d51762acf1c9-kube-api-access-pq24q\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m757h\" (UID: \"183a9d7f-d4be-4e9d-9cbd-d51762acf1c9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m757h" Aug 04 16:11:13.387896 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:11:13.387870 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/183a9d7f-d4be-4e9d-9cbd-d51762acf1c9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m757h\" (UID: \"183a9d7f-d4be-4e9d-9cbd-d51762acf1c9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m757h" Aug 04 16:11:13.387953 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:11:13.387908 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/183a9d7f-d4be-4e9d-9cbd-d51762acf1c9-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m757h\" (UID: \"183a9d7f-d4be-4e9d-9cbd-d51762acf1c9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m757h" Aug 04 16:11:13.388005 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:11:13.387971 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/183a9d7f-d4be-4e9d-9cbd-d51762acf1c9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m757h\" (UID: \"183a9d7f-d4be-4e9d-9cbd-d51762acf1c9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m757h" Aug 04 16:11:13.389801 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:11:13.389784 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/183a9d7f-d4be-4e9d-9cbd-d51762acf1c9-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m757h\" (UID: \"183a9d7f-d4be-4e9d-9cbd-d51762acf1c9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m757h" Aug 04 16:11:13.389973 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:11:13.389955 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/183a9d7f-d4be-4e9d-9cbd-d51762acf1c9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m757h\" (UID: \"183a9d7f-d4be-4e9d-9cbd-d51762acf1c9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m757h" Aug 04 16:11:13.396248 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:11:13.396227 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq24q\" (UniqueName: \"kubernetes.io/projected/183a9d7f-d4be-4e9d-9cbd-d51762acf1c9-kube-api-access-pq24q\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m757h\" (UID: \"183a9d7f-d4be-4e9d-9cbd-d51762acf1c9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m757h" Aug 04 16:11:13.496451 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:11:13.496410 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m757h" Aug 04 16:11:13.637396 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:11:13.637287 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-m757h"] Aug 04 16:11:13.640205 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:11:13.640175 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod183a9d7f_d4be_4e9d_9cbd_d51762acf1c9.slice/crio-e2f39d6ca28cf12b2c0a90a2fad65e64a45bce7d3a2a458efb3f9790be838e28 WatchSource:0}: Error finding container e2f39d6ca28cf12b2c0a90a2fad65e64a45bce7d3a2a458efb3f9790be838e28: Status 404 returned error can't find the container with id e2f39d6ca28cf12b2c0a90a2fad65e64a45bce7d3a2a458efb3f9790be838e28 Aug 04 16:11:14.330869 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:11:14.330834 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m757h" event={"ID":"183a9d7f-d4be-4e9d-9cbd-d51762acf1c9","Type":"ContainerStarted","Data":"6691e30ecc637664d0e64c3f17cf369b8111a2c58f1d60e478e97189a1dce5ed"} Aug 04 16:11:14.330869 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:11:14.330873 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m757h" event={"ID":"183a9d7f-d4be-4e9d-9cbd-d51762acf1c9","Type":"ContainerStarted","Data":"e2f39d6ca28cf12b2c0a90a2fad65e64a45bce7d3a2a458efb3f9790be838e28"} Aug 04 16:11:14.348765 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:11:14.348712 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m757h" podStartSLOduration=1.121886763 podStartE2EDuration="1.348694522s" totalImagesPullingTime="226.807759ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:11:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:11:14.347889902 +0000 UTC m=+873.765432029" watchObservedRunningTime="2026-08-04 16:11:14.348694522 +0000 UTC m=+873.766236648" Aug 04 16:11:23.496749 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:11:23.496708 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m757h" Aug 04 16:11:23.497164 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:11:23.496796 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m757h" Aug 04 16:11:23.509943 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:11:23.509915 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m757h" Aug 04 16:11:24.387782 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:11:24.387759 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m757h" Aug 04 16:11:40.993780 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:11:40.993753 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-gvb4l_c16f82ab-8340-49ee-b41e-5685d4592fd2/console-operator/2.log" Aug 04 16:11:40.995217 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:11:40.995197 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-gvb4l_c16f82ab-8340-49ee-b41e-5685d4592fd2/console-operator/2.log" Aug 04 16:11:40.997972 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:11:40.997950 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mtbc5_11f9cbee-a773-420b-9873-5c8cb7679dd8/ovn-acl-logging/0.log" Aug 04 16:11:40.999432 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:11:40.999413 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mtbc5_11f9cbee-a773-420b-9873-5c8cb7679dd8/ovn-acl-logging/0.log" Aug 04 16:12:06.548723 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:12:06.548690 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b4b599967-lxh6z"] Aug 04 16:12:06.551912 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:12:06.551898 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b4b599967-lxh6z" Aug 04 16:12:06.554783 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:12:06.554763 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Aug 04 16:12:06.554925 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:12:06.554771 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rv57c\"" Aug 04 16:12:06.560586 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:12:06.560544 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b4b599967-lxh6z"] Aug 04 16:12:06.630703 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:12:06.630666 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cf8c329c-cf6b-4ac0-bb3f-56c9d0b78597-tls-cert\") pod \"authorino-b4b599967-lxh6z\" (UID: \"cf8c329c-cf6b-4ac0-bb3f-56c9d0b78597\") " pod="kuadrant-system/authorino-b4b599967-lxh6z" Aug 04 16:12:06.630891 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:12:06.630728 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4zx4\" (UniqueName: \"kubernetes.io/projected/cf8c329c-cf6b-4ac0-bb3f-56c9d0b78597-kube-api-access-q4zx4\") pod \"authorino-b4b599967-lxh6z\" (UID: \"cf8c329c-cf6b-4ac0-bb3f-56c9d0b78597\") " pod="kuadrant-system/authorino-b4b599967-lxh6z" Aug 04 16:12:06.731405 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:12:06.731371 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4zx4\" (UniqueName: \"kubernetes.io/projected/cf8c329c-cf6b-4ac0-bb3f-56c9d0b78597-kube-api-access-q4zx4\") pod \"authorino-b4b599967-lxh6z\" (UID: \"cf8c329c-cf6b-4ac0-bb3f-56c9d0b78597\") " pod="kuadrant-system/authorino-b4b599967-lxh6z" Aug 04 16:12:06.731599 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:12:06.731431 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cf8c329c-cf6b-4ac0-bb3f-56c9d0b78597-tls-cert\") pod \"authorino-b4b599967-lxh6z\" (UID: \"cf8c329c-cf6b-4ac0-bb3f-56c9d0b78597\") " pod="kuadrant-system/authorino-b4b599967-lxh6z" Aug 04 16:12:06.733861 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:12:06.733839 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cf8c329c-cf6b-4ac0-bb3f-56c9d0b78597-tls-cert\") pod \"authorino-b4b599967-lxh6z\" (UID: \"cf8c329c-cf6b-4ac0-bb3f-56c9d0b78597\") " pod="kuadrant-system/authorino-b4b599967-lxh6z" Aug 04 16:12:06.740022 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:12:06.739998 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4zx4\" (UniqueName: \"kubernetes.io/projected/cf8c329c-cf6b-4ac0-bb3f-56c9d0b78597-kube-api-access-q4zx4\") pod \"authorino-b4b599967-lxh6z\" (UID: \"cf8c329c-cf6b-4ac0-bb3f-56c9d0b78597\") " pod="kuadrant-system/authorino-b4b599967-lxh6z" Aug 04 16:12:06.862500 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:12:06.862416 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b4b599967-lxh6z" Aug 04 16:12:06.996755 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:12:06.996721 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b4b599967-lxh6z"] Aug 04 16:12:06.999666 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:12:06.999638 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf8c329c_cf6b_4ac0_bb3f_56c9d0b78597.slice/crio-0540a29622f00e3473226323842a6fd1eab22cca9d2bd25a5fc91c75f1b83882 WatchSource:0}: Error finding container 0540a29622f00e3473226323842a6fd1eab22cca9d2bd25a5fc91c75f1b83882: Status 404 returned error can't find the container with id 0540a29622f00e3473226323842a6fd1eab22cca9d2bd25a5fc91c75f1b83882 Aug 04 16:12:07.548330 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:12:07.548293 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b4b599967-lxh6z" event={"ID":"cf8c329c-cf6b-4ac0-bb3f-56c9d0b78597","Type":"ContainerStarted","Data":"c5f8ead3f91f315abcc4559d7f33f4281bfac83eb17464bd3ce569bd66928764"} Aug 04 16:12:07.548518 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:12:07.548341 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b4b599967-lxh6z" event={"ID":"cf8c329c-cf6b-4ac0-bb3f-56c9d0b78597","Type":"ContainerStarted","Data":"0540a29622f00e3473226323842a6fd1eab22cca9d2bd25a5fc91c75f1b83882"} Aug 04 16:12:07.567075 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:12:07.566991 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b4b599967-lxh6z" podStartSLOduration=1.123176627 podStartE2EDuration="1.566968184s" totalImagesPullingTime="443.791557ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:12:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:12:07.564495429 +0000 UTC m=+926.982037562" watchObservedRunningTime="2026-08-04 16:12:07.566968184 +0000 UTC m=+926.984510314" Aug 04 16:12:56.795291 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:12:56.795253 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d474f94c9-kgcpp"] Aug 04 16:12:56.795818 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:12:56.795471 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-d474f94c9-kgcpp" podUID="83bfb81e-bf2d-468c-948a-3e5b25829dfb" containerName="manager" containerID="cri-o://f0ef97effe939cbeecd5de39c7e9bc2a42c33aa902137d0daa27cb1e11f0ef1e" gracePeriod=30 Aug 04 16:12:57.039976 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:12:57.039942 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d474f94c9-kgcpp" Aug 04 16:12:57.158482 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:12:57.158388 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnnwr\" (UniqueName: \"kubernetes.io/projected/83bfb81e-bf2d-468c-948a-3e5b25829dfb-kube-api-access-fnnwr\") pod \"83bfb81e-bf2d-468c-948a-3e5b25829dfb\" (UID: \"83bfb81e-bf2d-468c-948a-3e5b25829dfb\") " Aug 04 16:12:57.158650 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:12:57.158497 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/83bfb81e-bf2d-468c-948a-3e5b25829dfb-webhook-cert\") pod \"83bfb81e-bf2d-468c-948a-3e5b25829dfb\" (UID: \"83bfb81e-bf2d-468c-948a-3e5b25829dfb\") " Aug 04 16:12:57.160687 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:12:57.160661 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83bfb81e-bf2d-468c-948a-3e5b25829dfb-kube-api-access-fnnwr" (OuterVolumeSpecName: "kube-api-access-fnnwr") pod "83bfb81e-bf2d-468c-948a-3e5b25829dfb" (UID: "83bfb81e-bf2d-468c-948a-3e5b25829dfb"). InnerVolumeSpecName "kube-api-access-fnnwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:12:57.160800 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:12:57.160685 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83bfb81e-bf2d-468c-948a-3e5b25829dfb-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "83bfb81e-bf2d-468c-948a-3e5b25829dfb" (UID: "83bfb81e-bf2d-468c-948a-3e5b25829dfb"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:12:57.259853 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:12:57.259821 2573 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/83bfb81e-bf2d-468c-948a-3e5b25829dfb-webhook-cert\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:12:57.259853 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:12:57.259850 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnnwr\" (UniqueName: \"kubernetes.io/projected/83bfb81e-bf2d-468c-948a-3e5b25829dfb-kube-api-access-fnnwr\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:12:57.759253 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:12:57.759220 2573 generic.go:356] "Generic (PLEG): container finished" podID="83bfb81e-bf2d-468c-948a-3e5b25829dfb" containerID="f0ef97effe939cbeecd5de39c7e9bc2a42c33aa902137d0daa27cb1e11f0ef1e" exitCode=0 Aug 04 16:12:57.759430 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:12:57.759276 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d474f94c9-kgcpp" Aug 04 16:12:57.759430 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:12:57.759288 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d474f94c9-kgcpp" event={"ID":"83bfb81e-bf2d-468c-948a-3e5b25829dfb","Type":"ContainerDied","Data":"f0ef97effe939cbeecd5de39c7e9bc2a42c33aa902137d0daa27cb1e11f0ef1e"} Aug 04 16:12:57.759430 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:12:57.759329 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d474f94c9-kgcpp" event={"ID":"83bfb81e-bf2d-468c-948a-3e5b25829dfb","Type":"ContainerDied","Data":"edea0f01b921632caf3f80834cc23798c6b85dbb0f0f9b15fe3239e0e9cc9efb"} Aug 04 16:12:57.759430 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:12:57.759350 2573 scope.go:117] "RemoveContainer" containerID="f0ef97effe939cbeecd5de39c7e9bc2a42c33aa902137d0daa27cb1e11f0ef1e" Aug 04 16:12:57.768157 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:12:57.768141 2573 scope.go:117] "RemoveContainer" containerID="f0ef97effe939cbeecd5de39c7e9bc2a42c33aa902137d0daa27cb1e11f0ef1e" Aug 04 16:12:57.768376 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:12:57.768360 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0ef97effe939cbeecd5de39c7e9bc2a42c33aa902137d0daa27cb1e11f0ef1e\": container with ID starting with f0ef97effe939cbeecd5de39c7e9bc2a42c33aa902137d0daa27cb1e11f0ef1e not found: ID does not exist" containerID="f0ef97effe939cbeecd5de39c7e9bc2a42c33aa902137d0daa27cb1e11f0ef1e" Aug 04 16:12:57.768425 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:12:57.768384 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0ef97effe939cbeecd5de39c7e9bc2a42c33aa902137d0daa27cb1e11f0ef1e"} err="failed to get container status \"f0ef97effe939cbeecd5de39c7e9bc2a42c33aa902137d0daa27cb1e11f0ef1e\": rpc error: code = NotFound desc = could not find container \"f0ef97effe939cbeecd5de39c7e9bc2a42c33aa902137d0daa27cb1e11f0ef1e\": container with ID starting with f0ef97effe939cbeecd5de39c7e9bc2a42c33aa902137d0daa27cb1e11f0ef1e not found: ID does not exist" Aug 04 16:12:57.783103 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:12:57.783075 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d474f94c9-kgcpp"] Aug 04 16:12:57.789332 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:12:57.789304 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-d474f94c9-kgcpp"] Aug 04 16:12:59.078171 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:12:59.078138 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83bfb81e-bf2d-468c-948a-3e5b25829dfb" path="/var/lib/kubelet/pods/83bfb81e-bf2d-468c-948a-3e5b25829dfb/volumes" Aug 04 16:16:41.018479 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:16:41.018393 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-gvb4l_c16f82ab-8340-49ee-b41e-5685d4592fd2/console-operator/2.log" Aug 04 16:16:41.020434 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:16:41.020414 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-gvb4l_c16f82ab-8340-49ee-b41e-5685d4592fd2/console-operator/2.log" Aug 04 16:16:41.022700 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:16:41.022677 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mtbc5_11f9cbee-a773-420b-9873-5c8cb7679dd8/ovn-acl-logging/0.log" Aug 04 16:16:41.024675 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:16:41.024656 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mtbc5_11f9cbee-a773-420b-9873-5c8cb7679dd8/ovn-acl-logging/0.log" Aug 04 16:21:41.044321 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:21:41.044294 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-gvb4l_c16f82ab-8340-49ee-b41e-5685d4592fd2/console-operator/2.log" Aug 04 16:21:41.044912 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:21:41.044446 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-gvb4l_c16f82ab-8340-49ee-b41e-5685d4592fd2/console-operator/2.log" Aug 04 16:21:41.048197 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:21:41.048173 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mtbc5_11f9cbee-a773-420b-9873-5c8cb7679dd8/ovn-acl-logging/0.log" Aug 04 16:21:41.048337 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:21:41.048286 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mtbc5_11f9cbee-a773-420b-9873-5c8cb7679dd8/ovn-acl-logging/0.log" Aug 04 16:26:41.067407 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:26:41.067373 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-gvb4l_c16f82ab-8340-49ee-b41e-5685d4592fd2/console-operator/2.log" Aug 04 16:26:41.068509 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:26:41.068487 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-gvb4l_c16f82ab-8340-49ee-b41e-5685d4592fd2/console-operator/2.log" Aug 04 16:26:41.071190 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:26:41.071170 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mtbc5_11f9cbee-a773-420b-9873-5c8cb7679dd8/ovn-acl-logging/0.log" Aug 04 16:26:41.072375 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:26:41.072354 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mtbc5_11f9cbee-a773-420b-9873-5c8cb7679dd8/ovn-acl-logging/0.log" Aug 04 16:30:13.415830 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:13.415798 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq"] Aug 04 16:30:13.416270 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:13.416101 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83bfb81e-bf2d-468c-948a-3e5b25829dfb" containerName="manager" Aug 04 16:30:13.416270 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:13.416112 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="83bfb81e-bf2d-468c-948a-3e5b25829dfb" containerName="manager" Aug 04 16:30:13.416270 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:13.416176 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="83bfb81e-bf2d-468c-948a-3e5b25829dfb" containerName="manager" Aug 04 16:30:13.419335 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:13.419313 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq" Aug 04 16:30:13.421874 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:13.421845 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3a345e2e-serving-cert\"" Aug 04 16:30:13.430391 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:13.430366 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq"] Aug 04 16:30:13.538447 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:13.538402 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76db9fb7-f327-49f6-92c5-4344488d28d1-openshift-service-ca\") pod \"maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq\" (UID: \"76db9fb7-f327-49f6-92c5-4344488d28d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq" Aug 04 16:30:13.538673 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:13.538515 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvxqc\" (UniqueName: \"kubernetes.io/projected/76db9fb7-f327-49f6-92c5-4344488d28d1-kube-api-access-mvxqc\") pod \"maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq\" (UID: \"76db9fb7-f327-49f6-92c5-4344488d28d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq" Aug 04 16:30:13.538673 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:13.538592 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76db9fb7-f327-49f6-92c5-4344488d28d1-maas-api-tls\") pod \"maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq\" (UID: \"76db9fb7-f327-49f6-92c5-4344488d28d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq" Aug 04 16:30:13.639235 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:13.639194 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvxqc\" (UniqueName: \"kubernetes.io/projected/76db9fb7-f327-49f6-92c5-4344488d28d1-kube-api-access-mvxqc\") pod \"maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq\" (UID: \"76db9fb7-f327-49f6-92c5-4344488d28d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq" Aug 04 16:30:13.639413 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:13.639250 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76db9fb7-f327-49f6-92c5-4344488d28d1-maas-api-tls\") pod \"maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq\" (UID: \"76db9fb7-f327-49f6-92c5-4344488d28d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq" Aug 04 16:30:13.639413 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:13.639301 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76db9fb7-f327-49f6-92c5-4344488d28d1-openshift-service-ca\") pod \"maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq\" (UID: \"76db9fb7-f327-49f6-92c5-4344488d28d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq" Aug 04 16:30:13.639490 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:30:13.639415 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-3a345e2e-serving-cert: secret "maas-api-e2e-ait-3a345e2e-serving-cert" not found Aug 04 16:30:13.639524 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:30:13.639495 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76db9fb7-f327-49f6-92c5-4344488d28d1-maas-api-tls podName:76db9fb7-f327-49f6-92c5-4344488d28d1 nodeName:}" failed. No retries permitted until 2026-08-04 16:30:14.139474381 +0000 UTC m=+2013.557016486 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/76db9fb7-f327-49f6-92c5-4344488d28d1-maas-api-tls") pod "maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq" (UID: "76db9fb7-f327-49f6-92c5-4344488d28d1") : secret "maas-api-e2e-ait-3a345e2e-serving-cert" not found Aug 04 16:30:13.640015 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:13.639996 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76db9fb7-f327-49f6-92c5-4344488d28d1-openshift-service-ca\") pod \"maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq\" (UID: \"76db9fb7-f327-49f6-92c5-4344488d28d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq" Aug 04 16:30:13.648263 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:13.648238 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvxqc\" (UniqueName: \"kubernetes.io/projected/76db9fb7-f327-49f6-92c5-4344488d28d1-kube-api-access-mvxqc\") pod \"maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq\" (UID: \"76db9fb7-f327-49f6-92c5-4344488d28d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq" Aug 04 16:30:14.144184 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:14.144150 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76db9fb7-f327-49f6-92c5-4344488d28d1-maas-api-tls\") pod \"maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq\" (UID: \"76db9fb7-f327-49f6-92c5-4344488d28d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq" Aug 04 16:30:14.146714 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:14.146689 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76db9fb7-f327-49f6-92c5-4344488d28d1-maas-api-tls\") pod \"maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq\" (UID: \"76db9fb7-f327-49f6-92c5-4344488d28d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq" Aug 04 16:30:14.330247 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:14.330216 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq" Aug 04 16:30:14.463849 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:14.463766 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq"] Aug 04 16:30:14.466337 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:30:14.466304 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76db9fb7_f327_49f6_92c5_4344488d28d1.slice/crio-7b1b877f006c389bb0090ecadd5ab17663713613f5511c575013d21e4b9cc70e WatchSource:0}: Error finding container 7b1b877f006c389bb0090ecadd5ab17663713613f5511c575013d21e4b9cc70e: Status 404 returned error can't find the container with id 7b1b877f006c389bb0090ecadd5ab17663713613f5511c575013d21e4b9cc70e Aug 04 16:30:14.467634 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:14.467616 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 16:30:15.158627 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:15.158591 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq" event={"ID":"76db9fb7-f327-49f6-92c5-4344488d28d1","Type":"ContainerStarted","Data":"18571ce0641a13f274b9356fd0cbca36011be0bad52c9796c007705ebe26b7dc"} Aug 04 16:30:15.158815 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:15.158649 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq" event={"ID":"76db9fb7-f327-49f6-92c5-4344488d28d1","Type":"ContainerStarted","Data":"7b1b877f006c389bb0090ecadd5ab17663713613f5511c575013d21e4b9cc70e"} Aug 04 16:30:15.158815 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:15.158676 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq" Aug 04 16:30:15.175927 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:15.175873 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq" podStartSLOduration=1.715878654 podStartE2EDuration="2.175856396s" totalImagesPullingTime="459.977742ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:30:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:30:15.17397387 +0000 UTC m=+2014.591516007" watchObservedRunningTime="2026-08-04 16:30:15.175856396 +0000 UTC m=+2014.593398522" Aug 04 16:30:21.168879 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:21.168848 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq" Aug 04 16:30:53.931935 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:53.931897 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq"] Aug 04 16:30:53.932540 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:53.932124 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq" podUID="76db9fb7-f327-49f6-92c5-4344488d28d1" containerName="maas-api" containerID="cri-o://18571ce0641a13f274b9356fd0cbca36011be0bad52c9796c007705ebe26b7dc" gracePeriod=30 Aug 04 16:30:54.175500 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:54.175468 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq" Aug 04 16:30:54.279684 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:54.279653 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76db9fb7-f327-49f6-92c5-4344488d28d1-maas-api-tls\") pod \"76db9fb7-f327-49f6-92c5-4344488d28d1\" (UID: \"76db9fb7-f327-49f6-92c5-4344488d28d1\") " Aug 04 16:30:54.279887 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:54.279737 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76db9fb7-f327-49f6-92c5-4344488d28d1-openshift-service-ca\") pod \"76db9fb7-f327-49f6-92c5-4344488d28d1\" (UID: \"76db9fb7-f327-49f6-92c5-4344488d28d1\") " Aug 04 16:30:54.279887 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:54.279785 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvxqc\" (UniqueName: \"kubernetes.io/projected/76db9fb7-f327-49f6-92c5-4344488d28d1-kube-api-access-mvxqc\") pod \"76db9fb7-f327-49f6-92c5-4344488d28d1\" (UID: \"76db9fb7-f327-49f6-92c5-4344488d28d1\") " Aug 04 16:30:54.280281 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:54.280251 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76db9fb7-f327-49f6-92c5-4344488d28d1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "76db9fb7-f327-49f6-92c5-4344488d28d1" (UID: "76db9fb7-f327-49f6-92c5-4344488d28d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:30:54.284411 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:54.284373 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76db9fb7-f327-49f6-92c5-4344488d28d1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "76db9fb7-f327-49f6-92c5-4344488d28d1" (UID: "76db9fb7-f327-49f6-92c5-4344488d28d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:30:54.286543 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:54.286500 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76db9fb7-f327-49f6-92c5-4344488d28d1-kube-api-access-mvxqc" (OuterVolumeSpecName: "kube-api-access-mvxqc") pod "76db9fb7-f327-49f6-92c5-4344488d28d1" (UID: "76db9fb7-f327-49f6-92c5-4344488d28d1"). InnerVolumeSpecName "kube-api-access-mvxqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:30:54.323070 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:54.323036 2573 generic.go:356] "Generic (PLEG): container finished" podID="76db9fb7-f327-49f6-92c5-4344488d28d1" containerID="18571ce0641a13f274b9356fd0cbca36011be0bad52c9796c007705ebe26b7dc" exitCode=0 Aug 04 16:30:54.323225 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:54.323078 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq" Aug 04 16:30:54.323225 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:54.323084 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq" event={"ID":"76db9fb7-f327-49f6-92c5-4344488d28d1","Type":"ContainerDied","Data":"18571ce0641a13f274b9356fd0cbca36011be0bad52c9796c007705ebe26b7dc"} Aug 04 16:30:54.323225 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:54.323130 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq" event={"ID":"76db9fb7-f327-49f6-92c5-4344488d28d1","Type":"ContainerDied","Data":"7b1b877f006c389bb0090ecadd5ab17663713613f5511c575013d21e4b9cc70e"} Aug 04 16:30:54.323225 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:54.323155 2573 scope.go:117] "RemoveContainer" containerID="18571ce0641a13f274b9356fd0cbca36011be0bad52c9796c007705ebe26b7dc" Aug 04 16:30:54.332307 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:54.332286 2573 scope.go:117] "RemoveContainer" containerID="18571ce0641a13f274b9356fd0cbca36011be0bad52c9796c007705ebe26b7dc" Aug 04 16:30:54.332528 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:30:54.332509 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18571ce0641a13f274b9356fd0cbca36011be0bad52c9796c007705ebe26b7dc\": container with ID starting with 18571ce0641a13f274b9356fd0cbca36011be0bad52c9796c007705ebe26b7dc not found: ID does not exist" containerID="18571ce0641a13f274b9356fd0cbca36011be0bad52c9796c007705ebe26b7dc" Aug 04 16:30:54.332615 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:54.332538 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18571ce0641a13f274b9356fd0cbca36011be0bad52c9796c007705ebe26b7dc"} err="failed to get container status \"18571ce0641a13f274b9356fd0cbca36011be0bad52c9796c007705ebe26b7dc\": rpc error: code = NotFound desc = could not find container \"18571ce0641a13f274b9356fd0cbca36011be0bad52c9796c007705ebe26b7dc\": container with ID starting with 18571ce0641a13f274b9356fd0cbca36011be0bad52c9796c007705ebe26b7dc not found: ID does not exist" Aug 04 16:30:54.345828 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:54.345804 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq"] Aug 04 16:30:54.348476 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:54.348450 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3a345e2e-57dcf9bb6c-pkgcq"] Aug 04 16:30:54.380402 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:54.380363 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76db9fb7-f327-49f6-92c5-4344488d28d1-maas-api-tls\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:30:54.380402 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:54.380403 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76db9fb7-f327-49f6-92c5-4344488d28d1-openshift-service-ca\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:30:54.380402 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:54.380414 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvxqc\" (UniqueName: \"kubernetes.io/projected/76db9fb7-f327-49f6-92c5-4344488d28d1-kube-api-access-mvxqc\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:30:55.079122 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:55.079090 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76db9fb7-f327-49f6-92c5-4344488d28d1" path="/var/lib/kubelet/pods/76db9fb7-f327-49f6-92c5-4344488d28d1/volumes" Aug 04 16:30:57.707982 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:57.707950 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-64639142-bc899c98c-x2m8l"] Aug 04 16:30:57.708347 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:57.708269 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76db9fb7-f327-49f6-92c5-4344488d28d1" containerName="maas-api" Aug 04 16:30:57.708347 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:57.708281 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="76db9fb7-f327-49f6-92c5-4344488d28d1" containerName="maas-api" Aug 04 16:30:57.708421 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:57.708367 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="76db9fb7-f327-49f6-92c5-4344488d28d1" containerName="maas-api" Aug 04 16:30:57.712723 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:57.712701 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64639142-bc899c98c-x2m8l" Aug 04 16:30:57.715089 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:57.715068 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-64639142-serving-cert\"" Aug 04 16:30:57.722208 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:57.722184 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-64639142-bc899c98c-x2m8l"] Aug 04 16:30:57.812119 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:57.812070 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k4f2\" (UniqueName: \"kubernetes.io/projected/643bb35e-3909-4877-a847-33016b66837b-kube-api-access-8k4f2\") pod \"maas-api-e2e-migrate-64639142-bc899c98c-x2m8l\" (UID: \"643bb35e-3909-4877-a847-33016b66837b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64639142-bc899c98c-x2m8l" Aug 04 16:30:57.812119 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:57.812129 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/643bb35e-3909-4877-a847-33016b66837b-openshift-service-ca\") pod \"maas-api-e2e-migrate-64639142-bc899c98c-x2m8l\" (UID: \"643bb35e-3909-4877-a847-33016b66837b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64639142-bc899c98c-x2m8l" Aug 04 16:30:57.812332 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:57.812202 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/643bb35e-3909-4877-a847-33016b66837b-maas-api-tls\") pod \"maas-api-e2e-migrate-64639142-bc899c98c-x2m8l\" (UID: \"643bb35e-3909-4877-a847-33016b66837b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64639142-bc899c98c-x2m8l" Aug 04 16:30:57.913139 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:57.913096 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/643bb35e-3909-4877-a847-33016b66837b-maas-api-tls\") pod \"maas-api-e2e-migrate-64639142-bc899c98c-x2m8l\" (UID: \"643bb35e-3909-4877-a847-33016b66837b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64639142-bc899c98c-x2m8l" Aug 04 16:30:57.913317 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:57.913175 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k4f2\" (UniqueName: \"kubernetes.io/projected/643bb35e-3909-4877-a847-33016b66837b-kube-api-access-8k4f2\") pod \"maas-api-e2e-migrate-64639142-bc899c98c-x2m8l\" (UID: \"643bb35e-3909-4877-a847-33016b66837b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64639142-bc899c98c-x2m8l" Aug 04 16:30:57.913317 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:57.913204 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/643bb35e-3909-4877-a847-33016b66837b-openshift-service-ca\") pod \"maas-api-e2e-migrate-64639142-bc899c98c-x2m8l\" (UID: \"643bb35e-3909-4877-a847-33016b66837b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64639142-bc899c98c-x2m8l" Aug 04 16:30:57.913317 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:30:57.913251 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-64639142-serving-cert: secret "maas-api-e2e-migrate-64639142-serving-cert" not found Aug 04 16:30:57.913441 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:30:57.913329 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/643bb35e-3909-4877-a847-33016b66837b-maas-api-tls podName:643bb35e-3909-4877-a847-33016b66837b nodeName:}" failed. No retries permitted until 2026-08-04 16:30:58.413311551 +0000 UTC m=+2057.830853655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/643bb35e-3909-4877-a847-33016b66837b-maas-api-tls") pod "maas-api-e2e-migrate-64639142-bc899c98c-x2m8l" (UID: "643bb35e-3909-4877-a847-33016b66837b") : secret "maas-api-e2e-migrate-64639142-serving-cert" not found Aug 04 16:30:57.913831 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:57.913813 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/643bb35e-3909-4877-a847-33016b66837b-openshift-service-ca\") pod \"maas-api-e2e-migrate-64639142-bc899c98c-x2m8l\" (UID: \"643bb35e-3909-4877-a847-33016b66837b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64639142-bc899c98c-x2m8l" Aug 04 16:30:57.925929 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:57.925904 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k4f2\" (UniqueName: \"kubernetes.io/projected/643bb35e-3909-4877-a847-33016b66837b-kube-api-access-8k4f2\") pod \"maas-api-e2e-migrate-64639142-bc899c98c-x2m8l\" (UID: \"643bb35e-3909-4877-a847-33016b66837b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64639142-bc899c98c-x2m8l" Aug 04 16:30:58.417526 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:58.417485 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/643bb35e-3909-4877-a847-33016b66837b-maas-api-tls\") pod \"maas-api-e2e-migrate-64639142-bc899c98c-x2m8l\" (UID: \"643bb35e-3909-4877-a847-33016b66837b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64639142-bc899c98c-x2m8l" Aug 04 16:30:58.420307 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:58.420273 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/643bb35e-3909-4877-a847-33016b66837b-maas-api-tls\") pod \"maas-api-e2e-migrate-64639142-bc899c98c-x2m8l\" (UID: \"643bb35e-3909-4877-a847-33016b66837b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64639142-bc899c98c-x2m8l" Aug 04 16:30:58.625203 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:58.625166 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64639142-bc899c98c-x2m8l" Aug 04 16:30:58.762144 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:58.762117 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-64639142-bc899c98c-x2m8l"] Aug 04 16:30:58.765017 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:30:58.764987 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod643bb35e_3909_4877_a847_33016b66837b.slice/crio-1e1982e762ddb8af71822801b180cf882312512e7b570bee027e3bf109424ac2 WatchSource:0}: Error finding container 1e1982e762ddb8af71822801b180cf882312512e7b570bee027e3bf109424ac2: Status 404 returned error can't find the container with id 1e1982e762ddb8af71822801b180cf882312512e7b570bee027e3bf109424ac2 Aug 04 16:30:59.348826 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:59.348788 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64639142-bc899c98c-x2m8l" event={"ID":"643bb35e-3909-4877-a847-33016b66837b","Type":"ContainerStarted","Data":"4802e6b1cce048f866437a1e7cb36131b639d09e5c06b4aa7a1b5875556df5e3"} Aug 04 16:30:59.349023 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:59.348846 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64639142-bc899c98c-x2m8l" event={"ID":"643bb35e-3909-4877-a847-33016b66837b","Type":"ContainerStarted","Data":"1e1982e762ddb8af71822801b180cf882312512e7b570bee027e3bf109424ac2"} Aug 04 16:30:59.349023 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:59.348869 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64639142-bc899c98c-x2m8l" Aug 04 16:30:59.365812 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:30:59.365760 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64639142-bc899c98c-x2m8l" podStartSLOduration=1.983800081 podStartE2EDuration="2.365743423s" totalImagesPullingTime="381.943342ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:30:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:30:59.365000921 +0000 UTC m=+2058.782543048" watchObservedRunningTime="2026-08-04 16:30:59.365743423 +0000 UTC m=+2058.783285548" Aug 04 16:31:05.359912 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:05.359829 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64639142-bc899c98c-x2m8l" Aug 04 16:31:29.169259 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:29.169224 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-64639142-bc899c98c-x2m8l"] Aug 04 16:31:29.169869 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:29.169479 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64639142-bc899c98c-x2m8l" podUID="643bb35e-3909-4877-a847-33016b66837b" containerName="maas-api" containerID="cri-o://4802e6b1cce048f866437a1e7cb36131b639d09e5c06b4aa7a1b5875556df5e3" gracePeriod=30 Aug 04 16:31:29.417386 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:29.417362 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64639142-bc899c98c-x2m8l" Aug 04 16:31:29.467387 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:29.467353 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8k4f2\" (UniqueName: \"kubernetes.io/projected/643bb35e-3909-4877-a847-33016b66837b-kube-api-access-8k4f2\") pod \"643bb35e-3909-4877-a847-33016b66837b\" (UID: \"643bb35e-3909-4877-a847-33016b66837b\") " Aug 04 16:31:29.467387 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:29.467388 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/643bb35e-3909-4877-a847-33016b66837b-openshift-service-ca\") pod \"643bb35e-3909-4877-a847-33016b66837b\" (UID: \"643bb35e-3909-4877-a847-33016b66837b\") " Aug 04 16:31:29.467662 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:29.467409 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/643bb35e-3909-4877-a847-33016b66837b-maas-api-tls\") pod \"643bb35e-3909-4877-a847-33016b66837b\" (UID: \"643bb35e-3909-4877-a847-33016b66837b\") " Aug 04 16:31:29.468331 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:29.467973 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/643bb35e-3909-4877-a847-33016b66837b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "643bb35e-3909-4877-a847-33016b66837b" (UID: "643bb35e-3909-4877-a847-33016b66837b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:31:29.473065 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:29.470448 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/643bb35e-3909-4877-a847-33016b66837b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "643bb35e-3909-4877-a847-33016b66837b" (UID: "643bb35e-3909-4877-a847-33016b66837b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:31:29.473065 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:29.470463 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/643bb35e-3909-4877-a847-33016b66837b-kube-api-access-8k4f2" (OuterVolumeSpecName: "kube-api-access-8k4f2") pod "643bb35e-3909-4877-a847-33016b66837b" (UID: "643bb35e-3909-4877-a847-33016b66837b"). InnerVolumeSpecName "kube-api-access-8k4f2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:31:29.480762 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:29.480729 2573 generic.go:356] "Generic (PLEG): container finished" podID="643bb35e-3909-4877-a847-33016b66837b" containerID="4802e6b1cce048f866437a1e7cb36131b639d09e5c06b4aa7a1b5875556df5e3" exitCode=0 Aug 04 16:31:29.480893 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:29.480788 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64639142-bc899c98c-x2m8l" Aug 04 16:31:29.480893 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:29.480811 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64639142-bc899c98c-x2m8l" event={"ID":"643bb35e-3909-4877-a847-33016b66837b","Type":"ContainerDied","Data":"4802e6b1cce048f866437a1e7cb36131b639d09e5c06b4aa7a1b5875556df5e3"} Aug 04 16:31:29.481012 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:29.480908 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-64639142-bc899c98c-x2m8l" event={"ID":"643bb35e-3909-4877-a847-33016b66837b","Type":"ContainerDied","Data":"1e1982e762ddb8af71822801b180cf882312512e7b570bee027e3bf109424ac2"} Aug 04 16:31:29.481012 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:29.480941 2573 scope.go:117] "RemoveContainer" containerID="4802e6b1cce048f866437a1e7cb36131b639d09e5c06b4aa7a1b5875556df5e3" Aug 04 16:31:29.489807 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:29.489790 2573 scope.go:117] "RemoveContainer" containerID="4802e6b1cce048f866437a1e7cb36131b639d09e5c06b4aa7a1b5875556df5e3" Aug 04 16:31:29.490062 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:31:29.490043 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4802e6b1cce048f866437a1e7cb36131b639d09e5c06b4aa7a1b5875556df5e3\": container with ID starting with 4802e6b1cce048f866437a1e7cb36131b639d09e5c06b4aa7a1b5875556df5e3 not found: ID does not exist" containerID="4802e6b1cce048f866437a1e7cb36131b639d09e5c06b4aa7a1b5875556df5e3" Aug 04 16:31:29.490108 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:29.490070 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4802e6b1cce048f866437a1e7cb36131b639d09e5c06b4aa7a1b5875556df5e3"} err="failed to get container status \"4802e6b1cce048f866437a1e7cb36131b639d09e5c06b4aa7a1b5875556df5e3\": rpc error: code = NotFound desc = could not find container \"4802e6b1cce048f866437a1e7cb36131b639d09e5c06b4aa7a1b5875556df5e3\": container with ID starting with 4802e6b1cce048f866437a1e7cb36131b639d09e5c06b4aa7a1b5875556df5e3 not found: ID does not exist" Aug 04 16:31:29.507615 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:29.507577 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-64639142-bc899c98c-x2m8l"] Aug 04 16:31:29.509309 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:29.509279 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-64639142-bc899c98c-x2m8l"] Aug 04 16:31:29.569244 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:29.569212 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k4f2\" (UniqueName: \"kubernetes.io/projected/643bb35e-3909-4877-a847-33016b66837b-kube-api-access-8k4f2\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:31:29.569244 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:29.569239 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/643bb35e-3909-4877-a847-33016b66837b-openshift-service-ca\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:31:29.569244 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:29.569250 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/643bb35e-3909-4877-a847-33016b66837b-maas-api-tls\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:31:31.079927 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:31.079896 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="643bb35e-3909-4877-a847-33016b66837b" path="/var/lib/kubelet/pods/643bb35e-3909-4877-a847-33016b66837b/volumes" Aug 04 16:31:33.826275 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:33.826226 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1978158b-65d67479f4-hh9qf"] Aug 04 16:31:33.826770 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:33.826586 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="643bb35e-3909-4877-a847-33016b66837b" containerName="maas-api" Aug 04 16:31:33.826770 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:33.826598 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="643bb35e-3909-4877-a847-33016b66837b" containerName="maas-api" Aug 04 16:31:33.826770 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:33.826670 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="643bb35e-3909-4877-a847-33016b66837b" containerName="maas-api" Aug 04 16:31:33.831115 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:33.831094 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1978158b-65d67479f4-hh9qf" Aug 04 16:31:33.833361 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:33.833336 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1978158b-serving-cert\"" Aug 04 16:31:33.841533 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:33.841504 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1978158b-65d67479f4-hh9qf"] Aug 04 16:31:33.902949 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:33.902916 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c511319-b2f5-4b2a-ac49-d0ac91735597-maas-api-tls\") pod \"maas-api-e2e-ait-1978158b-65d67479f4-hh9qf\" (UID: \"8c511319-b2f5-4b2a-ac49-d0ac91735597\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1978158b-65d67479f4-hh9qf" Aug 04 16:31:33.903118 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:33.902958 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksdhj\" (UniqueName: \"kubernetes.io/projected/8c511319-b2f5-4b2a-ac49-d0ac91735597-kube-api-access-ksdhj\") pod \"maas-api-e2e-ait-1978158b-65d67479f4-hh9qf\" (UID: \"8c511319-b2f5-4b2a-ac49-d0ac91735597\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1978158b-65d67479f4-hh9qf" Aug 04 16:31:33.903118 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:33.903020 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c511319-b2f5-4b2a-ac49-d0ac91735597-openshift-service-ca\") pod \"maas-api-e2e-ait-1978158b-65d67479f4-hh9qf\" (UID: \"8c511319-b2f5-4b2a-ac49-d0ac91735597\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1978158b-65d67479f4-hh9qf" Aug 04 16:31:34.003691 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:34.003655 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c511319-b2f5-4b2a-ac49-d0ac91735597-maas-api-tls\") pod \"maas-api-e2e-ait-1978158b-65d67479f4-hh9qf\" (UID: \"8c511319-b2f5-4b2a-ac49-d0ac91735597\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1978158b-65d67479f4-hh9qf" Aug 04 16:31:34.003870 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:34.003700 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksdhj\" (UniqueName: \"kubernetes.io/projected/8c511319-b2f5-4b2a-ac49-d0ac91735597-kube-api-access-ksdhj\") pod \"maas-api-e2e-ait-1978158b-65d67479f4-hh9qf\" (UID: \"8c511319-b2f5-4b2a-ac49-d0ac91735597\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1978158b-65d67479f4-hh9qf" Aug 04 16:31:34.003870 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:34.003724 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c511319-b2f5-4b2a-ac49-d0ac91735597-openshift-service-ca\") pod \"maas-api-e2e-ait-1978158b-65d67479f4-hh9qf\" (UID: \"8c511319-b2f5-4b2a-ac49-d0ac91735597\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1978158b-65d67479f4-hh9qf" Aug 04 16:31:34.004302 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:34.004284 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c511319-b2f5-4b2a-ac49-d0ac91735597-openshift-service-ca\") pod \"maas-api-e2e-ait-1978158b-65d67479f4-hh9qf\" (UID: \"8c511319-b2f5-4b2a-ac49-d0ac91735597\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1978158b-65d67479f4-hh9qf" Aug 04 16:31:34.006226 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:34.006202 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c511319-b2f5-4b2a-ac49-d0ac91735597-maas-api-tls\") pod \"maas-api-e2e-ait-1978158b-65d67479f4-hh9qf\" (UID: \"8c511319-b2f5-4b2a-ac49-d0ac91735597\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1978158b-65d67479f4-hh9qf" Aug 04 16:31:34.013225 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:34.013197 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksdhj\" (UniqueName: \"kubernetes.io/projected/8c511319-b2f5-4b2a-ac49-d0ac91735597-kube-api-access-ksdhj\") pod \"maas-api-e2e-ait-1978158b-65d67479f4-hh9qf\" (UID: \"8c511319-b2f5-4b2a-ac49-d0ac91735597\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1978158b-65d67479f4-hh9qf" Aug 04 16:31:34.141606 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:34.141489 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1978158b-65d67479f4-hh9qf" Aug 04 16:31:34.278530 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:34.278507 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1978158b-65d67479f4-hh9qf"] Aug 04 16:31:34.280126 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:31:34.280098 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c511319_b2f5_4b2a_ac49_d0ac91735597.slice/crio-4f0aef8306aad504fa2160717cb0a00f30da7cfd8cd2cb132f2538965d827c9e WatchSource:0}: Error finding container 4f0aef8306aad504fa2160717cb0a00f30da7cfd8cd2cb132f2538965d827c9e: Status 404 returned error can't find the container with id 4f0aef8306aad504fa2160717cb0a00f30da7cfd8cd2cb132f2538965d827c9e Aug 04 16:31:34.504482 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:34.504443 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1978158b-65d67479f4-hh9qf" event={"ID":"8c511319-b2f5-4b2a-ac49-d0ac91735597","Type":"ContainerStarted","Data":"4f0aef8306aad504fa2160717cb0a00f30da7cfd8cd2cb132f2538965d827c9e"} Aug 04 16:31:36.514286 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:36.514239 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1978158b-65d67479f4-hh9qf" event={"ID":"8c511319-b2f5-4b2a-ac49-d0ac91735597","Type":"ContainerStarted","Data":"94cb9f749dbf08b9efc78b97ec81da5fed6251f6ed8e3e5e947cf1a774062c4e"} Aug 04 16:31:36.514754 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:36.514427 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1978158b-65d67479f4-hh9qf" Aug 04 16:31:36.531492 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:36.531445 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1978158b-65d67479f4-hh9qf" podStartSLOduration=1.938234306 podStartE2EDuration="3.531430664s" totalImagesPullingTime="1.593196358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:31:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:31:36.53030862 +0000 UTC m=+2095.947850746" watchObservedRunningTime="2026-08-04 16:31:36.531430664 +0000 UTC m=+2095.948972790" Aug 04 16:31:41.095825 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:41.095795 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-gvb4l_c16f82ab-8340-49ee-b41e-5685d4592fd2/console-operator/2.log" Aug 04 16:31:41.099017 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:41.098986 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-gvb4l_c16f82ab-8340-49ee-b41e-5685d4592fd2/console-operator/2.log" Aug 04 16:31:41.099703 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:41.099684 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mtbc5_11f9cbee-a773-420b-9873-5c8cb7679dd8/ovn-acl-logging/0.log" Aug 04 16:31:41.105793 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:41.105776 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mtbc5_11f9cbee-a773-420b-9873-5c8cb7679dd8/ovn-acl-logging/0.log" Aug 04 16:31:42.525180 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:31:42.525145 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1978158b-65d67479f4-hh9qf" Aug 04 16:32:04.440480 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:04.440443 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1978158b-65d67479f4-hh9qf"] Aug 04 16:32:04.441004 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:04.440695 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1978158b-65d67479f4-hh9qf" podUID="8c511319-b2f5-4b2a-ac49-d0ac91735597" containerName="maas-api" containerID="cri-o://94cb9f749dbf08b9efc78b97ec81da5fed6251f6ed8e3e5e947cf1a774062c4e" gracePeriod=30 Aug 04 16:32:04.634794 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:04.634750 2573 generic.go:356] "Generic (PLEG): container finished" podID="8c511319-b2f5-4b2a-ac49-d0ac91735597" containerID="94cb9f749dbf08b9efc78b97ec81da5fed6251f6ed8e3e5e947cf1a774062c4e" exitCode=0 Aug 04 16:32:04.634959 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:04.634846 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1978158b-65d67479f4-hh9qf" event={"ID":"8c511319-b2f5-4b2a-ac49-d0ac91735597","Type":"ContainerDied","Data":"94cb9f749dbf08b9efc78b97ec81da5fed6251f6ed8e3e5e947cf1a774062c4e"} Aug 04 16:32:04.693971 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:04.693914 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1978158b-65d67479f4-hh9qf" Aug 04 16:32:04.866608 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:04.866567 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c511319-b2f5-4b2a-ac49-d0ac91735597-maas-api-tls\") pod \"8c511319-b2f5-4b2a-ac49-d0ac91735597\" (UID: \"8c511319-b2f5-4b2a-ac49-d0ac91735597\") " Aug 04 16:32:04.866608 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:04.866611 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksdhj\" (UniqueName: \"kubernetes.io/projected/8c511319-b2f5-4b2a-ac49-d0ac91735597-kube-api-access-ksdhj\") pod \"8c511319-b2f5-4b2a-ac49-d0ac91735597\" (UID: \"8c511319-b2f5-4b2a-ac49-d0ac91735597\") " Aug 04 16:32:04.866868 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:04.866662 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c511319-b2f5-4b2a-ac49-d0ac91735597-openshift-service-ca\") pod \"8c511319-b2f5-4b2a-ac49-d0ac91735597\" (UID: \"8c511319-b2f5-4b2a-ac49-d0ac91735597\") " Aug 04 16:32:04.867053 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:04.867028 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c511319-b2f5-4b2a-ac49-d0ac91735597-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8c511319-b2f5-4b2a-ac49-d0ac91735597" (UID: "8c511319-b2f5-4b2a-ac49-d0ac91735597"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:32:04.868802 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:04.868772 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c511319-b2f5-4b2a-ac49-d0ac91735597-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8c511319-b2f5-4b2a-ac49-d0ac91735597" (UID: "8c511319-b2f5-4b2a-ac49-d0ac91735597"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:32:04.868906 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:04.868804 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c511319-b2f5-4b2a-ac49-d0ac91735597-kube-api-access-ksdhj" (OuterVolumeSpecName: "kube-api-access-ksdhj") pod "8c511319-b2f5-4b2a-ac49-d0ac91735597" (UID: "8c511319-b2f5-4b2a-ac49-d0ac91735597"). InnerVolumeSpecName "kube-api-access-ksdhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:32:04.967886 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:04.967860 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c511319-b2f5-4b2a-ac49-d0ac91735597-maas-api-tls\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:32:04.967886 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:04.967888 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksdhj\" (UniqueName: \"kubernetes.io/projected/8c511319-b2f5-4b2a-ac49-d0ac91735597-kube-api-access-ksdhj\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:32:04.967886 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:04.967898 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c511319-b2f5-4b2a-ac49-d0ac91735597-openshift-service-ca\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:32:05.640249 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:05.640214 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1978158b-65d67479f4-hh9qf" event={"ID":"8c511319-b2f5-4b2a-ac49-d0ac91735597","Type":"ContainerDied","Data":"4f0aef8306aad504fa2160717cb0a00f30da7cfd8cd2cb132f2538965d827c9e"} Aug 04 16:32:05.640688 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:05.640261 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1978158b-65d67479f4-hh9qf" Aug 04 16:32:05.640688 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:05.640304 2573 scope.go:117] "RemoveContainer" containerID="94cb9f749dbf08b9efc78b97ec81da5fed6251f6ed8e3e5e947cf1a774062c4e" Aug 04 16:32:05.655652 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:05.655622 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1978158b-65d67479f4-hh9qf"] Aug 04 16:32:05.660385 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:05.660357 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1978158b-65d67479f4-hh9qf"] Aug 04 16:32:07.078785 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:07.078748 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c511319-b2f5-4b2a-ac49-d0ac91735597" path="/var/lib/kubelet/pods/8c511319-b2f5-4b2a-ac49-d0ac91735597/volumes" Aug 04 16:32:10.140589 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:10.140533 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0eb193d3-c85d7548c-4njm4"] Aug 04 16:32:10.141111 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:10.140878 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c511319-b2f5-4b2a-ac49-d0ac91735597" containerName="maas-api" Aug 04 16:32:10.141111 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:10.140890 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c511319-b2f5-4b2a-ac49-d0ac91735597" containerName="maas-api" Aug 04 16:32:10.141111 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:10.140943 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c511319-b2f5-4b2a-ac49-d0ac91735597" containerName="maas-api" Aug 04 16:32:10.145093 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:10.145074 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb193d3-c85d7548c-4njm4" Aug 04 16:32:10.147708 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:10.147688 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-0eb193d3-serving-cert\"" Aug 04 16:32:10.155204 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:10.155180 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0eb193d3-c85d7548c-4njm4"] Aug 04 16:32:10.316948 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:10.316905 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0da2ec9-29a9-42b2-8738-a9f59d23cd0b-openshift-service-ca\") pod \"maas-api-e2e-derive-0eb193d3-c85d7548c-4njm4\" (UID: \"c0da2ec9-29a9-42b2-8738-a9f59d23cd0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb193d3-c85d7548c-4njm4" Aug 04 16:32:10.316948 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:10.316950 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0da2ec9-29a9-42b2-8738-a9f59d23cd0b-maas-api-tls\") pod \"maas-api-e2e-derive-0eb193d3-c85d7548c-4njm4\" (UID: \"c0da2ec9-29a9-42b2-8738-a9f59d23cd0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb193d3-c85d7548c-4njm4" Aug 04 16:32:10.317200 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:10.317011 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n276b\" (UniqueName: \"kubernetes.io/projected/c0da2ec9-29a9-42b2-8738-a9f59d23cd0b-kube-api-access-n276b\") pod \"maas-api-e2e-derive-0eb193d3-c85d7548c-4njm4\" (UID: \"c0da2ec9-29a9-42b2-8738-a9f59d23cd0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb193d3-c85d7548c-4njm4" Aug 04 16:32:10.418325 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:10.418234 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0da2ec9-29a9-42b2-8738-a9f59d23cd0b-openshift-service-ca\") pod \"maas-api-e2e-derive-0eb193d3-c85d7548c-4njm4\" (UID: \"c0da2ec9-29a9-42b2-8738-a9f59d23cd0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb193d3-c85d7548c-4njm4" Aug 04 16:32:10.418325 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:10.418283 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0da2ec9-29a9-42b2-8738-a9f59d23cd0b-maas-api-tls\") pod \"maas-api-e2e-derive-0eb193d3-c85d7548c-4njm4\" (UID: \"c0da2ec9-29a9-42b2-8738-a9f59d23cd0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb193d3-c85d7548c-4njm4" Aug 04 16:32:10.418325 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:10.418315 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n276b\" (UniqueName: \"kubernetes.io/projected/c0da2ec9-29a9-42b2-8738-a9f59d23cd0b-kube-api-access-n276b\") pod \"maas-api-e2e-derive-0eb193d3-c85d7548c-4njm4\" (UID: \"c0da2ec9-29a9-42b2-8738-a9f59d23cd0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb193d3-c85d7548c-4njm4" Aug 04 16:32:10.418913 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:10.418892 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0da2ec9-29a9-42b2-8738-a9f59d23cd0b-openshift-service-ca\") pod \"maas-api-e2e-derive-0eb193d3-c85d7548c-4njm4\" (UID: \"c0da2ec9-29a9-42b2-8738-a9f59d23cd0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb193d3-c85d7548c-4njm4" Aug 04 16:32:10.420940 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:10.420917 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0da2ec9-29a9-42b2-8738-a9f59d23cd0b-maas-api-tls\") pod \"maas-api-e2e-derive-0eb193d3-c85d7548c-4njm4\" (UID: \"c0da2ec9-29a9-42b2-8738-a9f59d23cd0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb193d3-c85d7548c-4njm4" Aug 04 16:32:10.426845 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:10.426816 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n276b\" (UniqueName: \"kubernetes.io/projected/c0da2ec9-29a9-42b2-8738-a9f59d23cd0b-kube-api-access-n276b\") pod \"maas-api-e2e-derive-0eb193d3-c85d7548c-4njm4\" (UID: \"c0da2ec9-29a9-42b2-8738-a9f59d23cd0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb193d3-c85d7548c-4njm4" Aug 04 16:32:10.455282 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:10.455242 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb193d3-c85d7548c-4njm4" Aug 04 16:32:10.590134 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:10.590098 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0eb193d3-c85d7548c-4njm4"] Aug 04 16:32:10.592886 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:32:10.592860 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0da2ec9_29a9_42b2_8738_a9f59d23cd0b.slice/crio-6e06f2ddc7c66b7d7d5357ad4e9d97da03904797b4207137d55054fa7756ba6d WatchSource:0}: Error finding container 6e06f2ddc7c66b7d7d5357ad4e9d97da03904797b4207137d55054fa7756ba6d: Status 404 returned error can't find the container with id 6e06f2ddc7c66b7d7d5357ad4e9d97da03904797b4207137d55054fa7756ba6d Aug 04 16:32:10.662911 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:10.662863 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb193d3-c85d7548c-4njm4" event={"ID":"c0da2ec9-29a9-42b2-8738-a9f59d23cd0b","Type":"ContainerStarted","Data":"6e06f2ddc7c66b7d7d5357ad4e9d97da03904797b4207137d55054fa7756ba6d"} Aug 04 16:32:13.676058 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:13.676022 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb193d3-c85d7548c-4njm4" event={"ID":"c0da2ec9-29a9-42b2-8738-a9f59d23cd0b","Type":"ContainerStarted","Data":"195027424a6a46548ae101d8f78a578acb53bd6d5168160ec78b73851a9d9977"} Aug 04 16:32:13.676492 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:13.676116 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb193d3-c85d7548c-4njm4" Aug 04 16:32:13.699020 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:13.698964 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb193d3-c85d7548c-4njm4" podStartSLOduration=1.381293434 podStartE2EDuration="3.698945494s" totalImagesPullingTime="2.31765206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:32:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:32:13.697467947 +0000 UTC m=+2133.115010073" watchObservedRunningTime="2026-08-04 16:32:13.698945494 +0000 UTC m=+2133.116487620" Aug 04 16:32:19.685765 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:19.685737 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb193d3-c85d7548c-4njm4" Aug 04 16:32:40.580583 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:40.580474 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0eb193d3-c85d7548c-4njm4"] Aug 04 16:32:40.581184 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:40.580771 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb193d3-c85d7548c-4njm4" podUID="c0da2ec9-29a9-42b2-8738-a9f59d23cd0b" containerName="maas-api" containerID="cri-o://195027424a6a46548ae101d8f78a578acb53bd6d5168160ec78b73851a9d9977" gracePeriod=30 Aug 04 16:32:40.794960 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:40.794930 2573 generic.go:356] "Generic (PLEG): container finished" podID="c0da2ec9-29a9-42b2-8738-a9f59d23cd0b" containerID="195027424a6a46548ae101d8f78a578acb53bd6d5168160ec78b73851a9d9977" exitCode=0 Aug 04 16:32:40.795157 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:40.794974 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb193d3-c85d7548c-4njm4" event={"ID":"c0da2ec9-29a9-42b2-8738-a9f59d23cd0b","Type":"ContainerDied","Data":"195027424a6a46548ae101d8f78a578acb53bd6d5168160ec78b73851a9d9977"} Aug 04 16:32:40.819641 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:40.819617 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb193d3-c85d7548c-4njm4" Aug 04 16:32:40.972645 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:40.972617 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n276b\" (UniqueName: \"kubernetes.io/projected/c0da2ec9-29a9-42b2-8738-a9f59d23cd0b-kube-api-access-n276b\") pod \"c0da2ec9-29a9-42b2-8738-a9f59d23cd0b\" (UID: \"c0da2ec9-29a9-42b2-8738-a9f59d23cd0b\") " Aug 04 16:32:40.972855 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:40.972697 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0da2ec9-29a9-42b2-8738-a9f59d23cd0b-openshift-service-ca\") pod \"c0da2ec9-29a9-42b2-8738-a9f59d23cd0b\" (UID: \"c0da2ec9-29a9-42b2-8738-a9f59d23cd0b\") " Aug 04 16:32:40.972855 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:40.972789 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0da2ec9-29a9-42b2-8738-a9f59d23cd0b-maas-api-tls\") pod \"c0da2ec9-29a9-42b2-8738-a9f59d23cd0b\" (UID: \"c0da2ec9-29a9-42b2-8738-a9f59d23cd0b\") " Aug 04 16:32:40.973031 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:40.973003 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0da2ec9-29a9-42b2-8738-a9f59d23cd0b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c0da2ec9-29a9-42b2-8738-a9f59d23cd0b" (UID: "c0da2ec9-29a9-42b2-8738-a9f59d23cd0b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:32:40.974847 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:40.974814 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0da2ec9-29a9-42b2-8738-a9f59d23cd0b-kube-api-access-n276b" (OuterVolumeSpecName: "kube-api-access-n276b") pod "c0da2ec9-29a9-42b2-8738-a9f59d23cd0b" (UID: "c0da2ec9-29a9-42b2-8738-a9f59d23cd0b"). InnerVolumeSpecName "kube-api-access-n276b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:32:40.974944 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:40.974897 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0da2ec9-29a9-42b2-8738-a9f59d23cd0b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c0da2ec9-29a9-42b2-8738-a9f59d23cd0b" (UID: "c0da2ec9-29a9-42b2-8738-a9f59d23cd0b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:32:41.073934 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:41.073892 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0da2ec9-29a9-42b2-8738-a9f59d23cd0b-openshift-service-ca\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:32:41.073934 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:41.073930 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0da2ec9-29a9-42b2-8738-a9f59d23cd0b-maas-api-tls\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:32:41.074134 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:41.073946 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n276b\" (UniqueName: \"kubernetes.io/projected/c0da2ec9-29a9-42b2-8738-a9f59d23cd0b-kube-api-access-n276b\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:32:41.800656 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:41.800625 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb193d3-c85d7548c-4njm4" Aug 04 16:32:41.801080 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:41.800621 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0eb193d3-c85d7548c-4njm4" event={"ID":"c0da2ec9-29a9-42b2-8738-a9f59d23cd0b","Type":"ContainerDied","Data":"6e06f2ddc7c66b7d7d5357ad4e9d97da03904797b4207137d55054fa7756ba6d"} Aug 04 16:32:41.801080 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:41.800758 2573 scope.go:117] "RemoveContainer" containerID="195027424a6a46548ae101d8f78a578acb53bd6d5168160ec78b73851a9d9977" Aug 04 16:32:41.837850 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:41.837817 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0eb193d3-c85d7548c-4njm4"] Aug 04 16:32:41.843465 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:41.843432 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0eb193d3-c85d7548c-4njm4"] Aug 04 16:32:43.078979 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:32:43.078945 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0da2ec9-29a9-42b2-8738-a9f59d23cd0b" path="/var/lib/kubelet/pods/c0da2ec9-29a9-42b2-8738-a9f59d23cd0b/volumes" Aug 04 16:34:44.607617 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:34:44.607582 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c"] Aug 04 16:34:44.608136 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:34:44.608024 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0da2ec9-29a9-42b2-8738-a9f59d23cd0b" containerName="maas-api" Aug 04 16:34:44.608136 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:34:44.608044 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0da2ec9-29a9-42b2-8738-a9f59d23cd0b" containerName="maas-api" Aug 04 16:34:44.608254 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:34:44.608142 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0da2ec9-29a9-42b2-8738-a9f59d23cd0b" containerName="maas-api" Aug 04 16:34:44.611294 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:34:44.611271 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c" Aug 04 16:34:44.613824 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:34:44.613798 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7c9a38ca-serving-cert\"" Aug 04 16:34:44.623194 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:34:44.623166 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c"] Aug 04 16:34:44.626827 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:34:44.626804 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cea3088-165d-459d-99d7-c4a251c7d966-maas-api-tls\") pod \"maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c\" (UID: \"6cea3088-165d-459d-99d7-c4a251c7d966\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c" Aug 04 16:34:44.626976 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:34:44.626845 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgzb4\" (UniqueName: \"kubernetes.io/projected/6cea3088-165d-459d-99d7-c4a251c7d966-kube-api-access-jgzb4\") pod \"maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c\" (UID: \"6cea3088-165d-459d-99d7-c4a251c7d966\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c" Aug 04 16:34:44.626976 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:34:44.626873 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cea3088-165d-459d-99d7-c4a251c7d966-openshift-service-ca\") pod \"maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c\" (UID: \"6cea3088-165d-459d-99d7-c4a251c7d966\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c" Aug 04 16:34:44.727982 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:34:44.727942 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cea3088-165d-459d-99d7-c4a251c7d966-maas-api-tls\") pod \"maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c\" (UID: \"6cea3088-165d-459d-99d7-c4a251c7d966\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c" Aug 04 16:34:44.728218 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:34:44.728001 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgzb4\" (UniqueName: \"kubernetes.io/projected/6cea3088-165d-459d-99d7-c4a251c7d966-kube-api-access-jgzb4\") pod \"maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c\" (UID: \"6cea3088-165d-459d-99d7-c4a251c7d966\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c" Aug 04 16:34:44.728218 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:34:44.728049 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cea3088-165d-459d-99d7-c4a251c7d966-openshift-service-ca\") pod \"maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c\" (UID: \"6cea3088-165d-459d-99d7-c4a251c7d966\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c" Aug 04 16:34:44.728218 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:34:44.728087 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-7c9a38ca-serving-cert: secret "maas-api-e2e-mt-7c9a38ca-serving-cert" not found Aug 04 16:34:44.728218 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:34:44.728173 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cea3088-165d-459d-99d7-c4a251c7d966-maas-api-tls podName:6cea3088-165d-459d-99d7-c4a251c7d966 nodeName:}" failed. No retries permitted until 2026-08-04 16:34:45.228157155 +0000 UTC m=+2284.645699259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6cea3088-165d-459d-99d7-c4a251c7d966-maas-api-tls") pod "maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c" (UID: "6cea3088-165d-459d-99d7-c4a251c7d966") : secret "maas-api-e2e-mt-7c9a38ca-serving-cert" not found Aug 04 16:34:44.728757 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:34:44.728737 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cea3088-165d-459d-99d7-c4a251c7d966-openshift-service-ca\") pod \"maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c\" (UID: \"6cea3088-165d-459d-99d7-c4a251c7d966\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c" Aug 04 16:34:44.736768 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:34:44.736742 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgzb4\" (UniqueName: \"kubernetes.io/projected/6cea3088-165d-459d-99d7-c4a251c7d966-kube-api-access-jgzb4\") pod \"maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c\" (UID: \"6cea3088-165d-459d-99d7-c4a251c7d966\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c" Aug 04 16:34:45.231625 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:34:45.231592 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cea3088-165d-459d-99d7-c4a251c7d966-maas-api-tls\") pod \"maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c\" (UID: \"6cea3088-165d-459d-99d7-c4a251c7d966\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c" Aug 04 16:34:45.234058 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:34:45.234026 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cea3088-165d-459d-99d7-c4a251c7d966-maas-api-tls\") pod \"maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c\" (UID: \"6cea3088-165d-459d-99d7-c4a251c7d966\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c" Aug 04 16:34:45.522445 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:34:45.522353 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c" Aug 04 16:34:45.684841 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:34:45.684804 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c"] Aug 04 16:34:45.691184 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:34:45.691147 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cea3088_165d_459d_99d7_c4a251c7d966.slice/crio-df3bfabd0cf8f7973fa5740a6868e36503499a297cf8a48b1706fe8d5462e5b4 WatchSource:0}: Error finding container df3bfabd0cf8f7973fa5740a6868e36503499a297cf8a48b1706fe8d5462e5b4: Status 404 returned error can't find the container with id df3bfabd0cf8f7973fa5740a6868e36503499a297cf8a48b1706fe8d5462e5b4 Aug 04 16:34:46.331778 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:34:46.331742 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c" event={"ID":"6cea3088-165d-459d-99d7-c4a251c7d966","Type":"ContainerStarted","Data":"df3bfabd0cf8f7973fa5740a6868e36503499a297cf8a48b1706fe8d5462e5b4"} Aug 04 16:34:47.337024 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:34:47.336993 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c" event={"ID":"6cea3088-165d-459d-99d7-c4a251c7d966","Type":"ContainerStarted","Data":"140bc32f6d9037bdacf7eb6bfad25ed73823c019daa54c684548b42ff48e2b02"} Aug 04 16:34:47.337457 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:34:47.337108 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c" Aug 04 16:34:53.346055 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:34:53.346028 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c" Aug 04 16:34:53.370312 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:34:53.370257 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c" podStartSLOduration=8.493105103 podStartE2EDuration="9.37024022s" totalImagesPullingTime="877.135117ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:34:47.364013223 +0000 UTC m=+2286.781555360" watchObservedRunningTime="2026-08-04 16:34:53.37024022 +0000 UTC m=+2292.787782348" Aug 04 16:35:26.726091 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:35:26.726050 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c"] Aug 04 16:35:26.728721 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:35:26.726317 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c" podUID="6cea3088-165d-459d-99d7-c4a251c7d966" containerName="maas-api" containerID="cri-o://140bc32f6d9037bdacf7eb6bfad25ed73823c019daa54c684548b42ff48e2b02" gracePeriod=30 Aug 04 16:35:26.971002 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:35:26.970979 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c" Aug 04 16:35:27.075829 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:35:27.075746 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cea3088-165d-459d-99d7-c4a251c7d966-maas-api-tls\") pod \"6cea3088-165d-459d-99d7-c4a251c7d966\" (UID: \"6cea3088-165d-459d-99d7-c4a251c7d966\") " Aug 04 16:35:27.075829 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:35:27.075804 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgzb4\" (UniqueName: \"kubernetes.io/projected/6cea3088-165d-459d-99d7-c4a251c7d966-kube-api-access-jgzb4\") pod \"6cea3088-165d-459d-99d7-c4a251c7d966\" (UID: \"6cea3088-165d-459d-99d7-c4a251c7d966\") " Aug 04 16:35:27.076042 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:35:27.075845 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cea3088-165d-459d-99d7-c4a251c7d966-openshift-service-ca\") pod \"6cea3088-165d-459d-99d7-c4a251c7d966\" (UID: \"6cea3088-165d-459d-99d7-c4a251c7d966\") " Aug 04 16:35:27.076347 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:35:27.076313 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cea3088-165d-459d-99d7-c4a251c7d966-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6cea3088-165d-459d-99d7-c4a251c7d966" (UID: "6cea3088-165d-459d-99d7-c4a251c7d966"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:35:27.077929 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:35:27.077902 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cea3088-165d-459d-99d7-c4a251c7d966-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6cea3088-165d-459d-99d7-c4a251c7d966" (UID: "6cea3088-165d-459d-99d7-c4a251c7d966"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:35:27.078216 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:35:27.078192 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cea3088-165d-459d-99d7-c4a251c7d966-kube-api-access-jgzb4" (OuterVolumeSpecName: "kube-api-access-jgzb4") pod "6cea3088-165d-459d-99d7-c4a251c7d966" (UID: "6cea3088-165d-459d-99d7-c4a251c7d966"). InnerVolumeSpecName "kube-api-access-jgzb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:35:27.177131 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:35:27.177093 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cea3088-165d-459d-99d7-c4a251c7d966-maas-api-tls\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:35:27.177131 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:35:27.177126 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgzb4\" (UniqueName: \"kubernetes.io/projected/6cea3088-165d-459d-99d7-c4a251c7d966-kube-api-access-jgzb4\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:35:27.177338 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:35:27.177142 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cea3088-165d-459d-99d7-c4a251c7d966-openshift-service-ca\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:35:27.507326 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:35:27.507290 2573 generic.go:356] "Generic (PLEG): container finished" podID="6cea3088-165d-459d-99d7-c4a251c7d966" containerID="140bc32f6d9037bdacf7eb6bfad25ed73823c019daa54c684548b42ff48e2b02" exitCode=0 Aug 04 16:35:27.507508 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:35:27.507336 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c" Aug 04 16:35:27.507508 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:35:27.507380 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c" event={"ID":"6cea3088-165d-459d-99d7-c4a251c7d966","Type":"ContainerDied","Data":"140bc32f6d9037bdacf7eb6bfad25ed73823c019daa54c684548b42ff48e2b02"} Aug 04 16:35:27.507508 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:35:27.507445 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c" event={"ID":"6cea3088-165d-459d-99d7-c4a251c7d966","Type":"ContainerDied","Data":"df3bfabd0cf8f7973fa5740a6868e36503499a297cf8a48b1706fe8d5462e5b4"} Aug 04 16:35:27.507508 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:35:27.507476 2573 scope.go:117] "RemoveContainer" containerID="140bc32f6d9037bdacf7eb6bfad25ed73823c019daa54c684548b42ff48e2b02" Aug 04 16:35:27.515820 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:35:27.515800 2573 scope.go:117] "RemoveContainer" containerID="140bc32f6d9037bdacf7eb6bfad25ed73823c019daa54c684548b42ff48e2b02" Aug 04 16:35:27.516092 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:35:27.516075 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"140bc32f6d9037bdacf7eb6bfad25ed73823c019daa54c684548b42ff48e2b02\": container with ID starting with 140bc32f6d9037bdacf7eb6bfad25ed73823c019daa54c684548b42ff48e2b02 not found: ID does not exist" containerID="140bc32f6d9037bdacf7eb6bfad25ed73823c019daa54c684548b42ff48e2b02" Aug 04 16:35:27.516135 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:35:27.516100 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"140bc32f6d9037bdacf7eb6bfad25ed73823c019daa54c684548b42ff48e2b02"} err="failed to get container status \"140bc32f6d9037bdacf7eb6bfad25ed73823c019daa54c684548b42ff48e2b02\": rpc error: code = NotFound desc = could not find container \"140bc32f6d9037bdacf7eb6bfad25ed73823c019daa54c684548b42ff48e2b02\": container with ID starting with 140bc32f6d9037bdacf7eb6bfad25ed73823c019daa54c684548b42ff48e2b02 not found: ID does not exist" Aug 04 16:35:27.528623 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:35:27.528592 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c"] Aug 04 16:35:27.537781 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:35:27.537544 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7c9a38ca-7dc7ccbc97-9rh6c"] Aug 04 16:35:29.078232 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:35:29.078195 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cea3088-165d-459d-99d7-c4a251c7d966" path="/var/lib/kubelet/pods/6cea3088-165d-459d-99d7-c4a251c7d966/volumes" Aug 04 16:36:41.121412 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:36:41.121378 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-gvb4l_c16f82ab-8340-49ee-b41e-5685d4592fd2/console-operator/2.log" Aug 04 16:36:41.131519 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:36:41.131486 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mtbc5_11f9cbee-a773-420b-9873-5c8cb7679dd8/ovn-acl-logging/0.log" Aug 04 16:36:41.132205 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:36:41.132182 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-gvb4l_c16f82ab-8340-49ee-b41e-5685d4592fd2/console-operator/2.log" Aug 04 16:36:41.136412 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:36:41.136394 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mtbc5_11f9cbee-a773-420b-9873-5c8cb7679dd8/ovn-acl-logging/0.log" Aug 04 16:37:55.652719 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:37:55.652682 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c2a82a29-794697655d-65hcq"] Aug 04 16:37:55.653185 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:37:55.653024 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cea3088-165d-459d-99d7-c4a251c7d966" containerName="maas-api" Aug 04 16:37:55.653185 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:37:55.653038 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cea3088-165d-459d-99d7-c4a251c7d966" containerName="maas-api" Aug 04 16:37:55.653185 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:37:55.653097 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cea3088-165d-459d-99d7-c4a251c7d966" containerName="maas-api" Aug 04 16:37:55.656021 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:37:55.656006 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2a82a29-794697655d-65hcq" Aug 04 16:37:55.658717 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:37:55.658698 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c2a82a29-serving-cert\"" Aug 04 16:37:55.674661 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:37:55.674631 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c2a82a29-794697655d-65hcq"] Aug 04 16:37:55.768672 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:37:55.768634 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43913e0b-e433-4207-8df2-f651aa9491ce-maas-api-tls\") pod \"maas-api-e2e-mt-c2a82a29-794697655d-65hcq\" (UID: \"43913e0b-e433-4207-8df2-f651aa9491ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2a82a29-794697655d-65hcq" Aug 04 16:37:55.768854 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:37:55.768742 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43913e0b-e433-4207-8df2-f651aa9491ce-openshift-service-ca\") pod \"maas-api-e2e-mt-c2a82a29-794697655d-65hcq\" (UID: \"43913e0b-e433-4207-8df2-f651aa9491ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2a82a29-794697655d-65hcq" Aug 04 16:37:55.768854 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:37:55.768814 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j4wq\" (UniqueName: \"kubernetes.io/projected/43913e0b-e433-4207-8df2-f651aa9491ce-kube-api-access-8j4wq\") pod \"maas-api-e2e-mt-c2a82a29-794697655d-65hcq\" (UID: \"43913e0b-e433-4207-8df2-f651aa9491ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2a82a29-794697655d-65hcq" Aug 04 16:37:55.870180 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:37:55.870142 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43913e0b-e433-4207-8df2-f651aa9491ce-openshift-service-ca\") pod \"maas-api-e2e-mt-c2a82a29-794697655d-65hcq\" (UID: \"43913e0b-e433-4207-8df2-f651aa9491ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2a82a29-794697655d-65hcq" Aug 04 16:37:55.870357 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:37:55.870194 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j4wq\" (UniqueName: \"kubernetes.io/projected/43913e0b-e433-4207-8df2-f651aa9491ce-kube-api-access-8j4wq\") pod \"maas-api-e2e-mt-c2a82a29-794697655d-65hcq\" (UID: \"43913e0b-e433-4207-8df2-f651aa9491ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2a82a29-794697655d-65hcq" Aug 04 16:37:55.870357 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:37:55.870239 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43913e0b-e433-4207-8df2-f651aa9491ce-maas-api-tls\") pod \"maas-api-e2e-mt-c2a82a29-794697655d-65hcq\" (UID: \"43913e0b-e433-4207-8df2-f651aa9491ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2a82a29-794697655d-65hcq" Aug 04 16:37:55.870476 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:37:55.870358 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c2a82a29-serving-cert: secret "maas-api-e2e-mt-c2a82a29-serving-cert" not found Aug 04 16:37:55.870476 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:37:55.870418 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43913e0b-e433-4207-8df2-f651aa9491ce-maas-api-tls podName:43913e0b-e433-4207-8df2-f651aa9491ce nodeName:}" failed. No retries permitted until 2026-08-04 16:37:56.370398353 +0000 UTC m=+2475.787940456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/43913e0b-e433-4207-8df2-f651aa9491ce-maas-api-tls") pod "maas-api-e2e-mt-c2a82a29-794697655d-65hcq" (UID: "43913e0b-e433-4207-8df2-f651aa9491ce") : secret "maas-api-e2e-mt-c2a82a29-serving-cert" not found Aug 04 16:37:55.870965 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:37:55.870941 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43913e0b-e433-4207-8df2-f651aa9491ce-openshift-service-ca\") pod \"maas-api-e2e-mt-c2a82a29-794697655d-65hcq\" (UID: \"43913e0b-e433-4207-8df2-f651aa9491ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2a82a29-794697655d-65hcq" Aug 04 16:37:55.884675 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:37:55.884647 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j4wq\" (UniqueName: \"kubernetes.io/projected/43913e0b-e433-4207-8df2-f651aa9491ce-kube-api-access-8j4wq\") pod \"maas-api-e2e-mt-c2a82a29-794697655d-65hcq\" (UID: \"43913e0b-e433-4207-8df2-f651aa9491ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2a82a29-794697655d-65hcq" Aug 04 16:37:56.375936 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:37:56.375896 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43913e0b-e433-4207-8df2-f651aa9491ce-maas-api-tls\") pod \"maas-api-e2e-mt-c2a82a29-794697655d-65hcq\" (UID: \"43913e0b-e433-4207-8df2-f651aa9491ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2a82a29-794697655d-65hcq" Aug 04 16:37:56.378407 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:37:56.378380 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43913e0b-e433-4207-8df2-f651aa9491ce-maas-api-tls\") pod \"maas-api-e2e-mt-c2a82a29-794697655d-65hcq\" (UID: \"43913e0b-e433-4207-8df2-f651aa9491ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2a82a29-794697655d-65hcq" Aug 04 16:37:56.565801 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:37:56.565771 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2a82a29-794697655d-65hcq" Aug 04 16:37:56.703248 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:37:56.703210 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c2a82a29-794697655d-65hcq"] Aug 04 16:37:56.706174 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:37:56.706138 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43913e0b_e433_4207_8df2_f651aa9491ce.slice/crio-d1d0537c568e55a46baf06d2f3d82a4627ed071d4b1d0d4127fdfbb983ddb8e2 WatchSource:0}: Error finding container d1d0537c568e55a46baf06d2f3d82a4627ed071d4b1d0d4127fdfbb983ddb8e2: Status 404 returned error can't find the container with id d1d0537c568e55a46baf06d2f3d82a4627ed071d4b1d0d4127fdfbb983ddb8e2 Aug 04 16:37:56.707369 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:37:56.707350 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 16:37:57.140576 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:37:57.140530 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2a82a29-794697655d-65hcq" event={"ID":"43913e0b-e433-4207-8df2-f651aa9491ce","Type":"ContainerStarted","Data":"d1d0537c568e55a46baf06d2f3d82a4627ed071d4b1d0d4127fdfbb983ddb8e2"} Aug 04 16:37:58.151307 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:37:58.151265 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2a82a29-794697655d-65hcq" event={"ID":"43913e0b-e433-4207-8df2-f651aa9491ce","Type":"ContainerStarted","Data":"a9f3b526bda7db9e06966858f70719a303bd9038a102ff40fd31dcbf6764b6a0"} Aug 04 16:37:58.151801 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:37:58.151361 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2a82a29-794697655d-65hcq" Aug 04 16:37:58.170028 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:37:58.169983 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2a82a29-794697655d-65hcq" podStartSLOduration=2.6872696019999998 podStartE2EDuration="3.169968086s" totalImagesPullingTime="482.698484ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:37:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:37:58.168879895 +0000 UTC m=+2477.586422025" watchObservedRunningTime="2026-08-04 16:37:58.169968086 +0000 UTC m=+2477.587510211" Aug 04 16:38:04.161062 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:38:04.161033 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2a82a29-794697655d-65hcq" Aug 04 16:39:14.827393 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:39:14.827346 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c2a82a29-794697655d-65hcq"] Aug 04 16:39:14.827929 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:39:14.827571 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2a82a29-794697655d-65hcq" podUID="43913e0b-e433-4207-8df2-f651aa9491ce" containerName="maas-api" containerID="cri-o://a9f3b526bda7db9e06966858f70719a303bd9038a102ff40fd31dcbf6764b6a0" gracePeriod=30 Aug 04 16:39:15.071288 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:39:15.071261 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2a82a29-794697655d-65hcq" Aug 04 16:39:15.115246 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:39:15.115161 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43913e0b-e433-4207-8df2-f651aa9491ce-maas-api-tls\") pod \"43913e0b-e433-4207-8df2-f651aa9491ce\" (UID: \"43913e0b-e433-4207-8df2-f651aa9491ce\") " Aug 04 16:39:15.115411 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:39:15.115265 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43913e0b-e433-4207-8df2-f651aa9491ce-openshift-service-ca\") pod \"43913e0b-e433-4207-8df2-f651aa9491ce\" (UID: \"43913e0b-e433-4207-8df2-f651aa9491ce\") " Aug 04 16:39:15.115411 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:39:15.115295 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8j4wq\" (UniqueName: \"kubernetes.io/projected/43913e0b-e433-4207-8df2-f651aa9491ce-kube-api-access-8j4wq\") pod \"43913e0b-e433-4207-8df2-f651aa9491ce\" (UID: \"43913e0b-e433-4207-8df2-f651aa9491ce\") " Aug 04 16:39:15.115670 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:39:15.115644 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43913e0b-e433-4207-8df2-f651aa9491ce-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "43913e0b-e433-4207-8df2-f651aa9491ce" (UID: "43913e0b-e433-4207-8df2-f651aa9491ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:39:15.117325 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:39:15.117304 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43913e0b-e433-4207-8df2-f651aa9491ce-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "43913e0b-e433-4207-8df2-f651aa9491ce" (UID: "43913e0b-e433-4207-8df2-f651aa9491ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:39:15.117389 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:39:15.117360 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43913e0b-e433-4207-8df2-f651aa9491ce-kube-api-access-8j4wq" (OuterVolumeSpecName: "kube-api-access-8j4wq") pod "43913e0b-e433-4207-8df2-f651aa9491ce" (UID: "43913e0b-e433-4207-8df2-f651aa9491ce"). InnerVolumeSpecName "kube-api-access-8j4wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:39:15.216679 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:39:15.216642 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43913e0b-e433-4207-8df2-f651aa9491ce-maas-api-tls\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:39:15.216679 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:39:15.216673 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43913e0b-e433-4207-8df2-f651aa9491ce-openshift-service-ca\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:39:15.216679 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:39:15.216683 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j4wq\" (UniqueName: \"kubernetes.io/projected/43913e0b-e433-4207-8df2-f651aa9491ce-kube-api-access-8j4wq\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:39:15.475961 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:39:15.475926 2573 generic.go:356] "Generic (PLEG): container finished" podID="43913e0b-e433-4207-8df2-f651aa9491ce" containerID="a9f3b526bda7db9e06966858f70719a303bd9038a102ff40fd31dcbf6764b6a0" exitCode=0 Aug 04 16:39:15.476137 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:39:15.475978 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2a82a29-794697655d-65hcq" Aug 04 16:39:15.476137 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:39:15.475994 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2a82a29-794697655d-65hcq" event={"ID":"43913e0b-e433-4207-8df2-f651aa9491ce","Type":"ContainerDied","Data":"a9f3b526bda7db9e06966858f70719a303bd9038a102ff40fd31dcbf6764b6a0"} Aug 04 16:39:15.476137 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:39:15.476051 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2a82a29-794697655d-65hcq" event={"ID":"43913e0b-e433-4207-8df2-f651aa9491ce","Type":"ContainerDied","Data":"d1d0537c568e55a46baf06d2f3d82a4627ed071d4b1d0d4127fdfbb983ddb8e2"} Aug 04 16:39:15.476137 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:39:15.476081 2573 scope.go:117] "RemoveContainer" containerID="a9f3b526bda7db9e06966858f70719a303bd9038a102ff40fd31dcbf6764b6a0" Aug 04 16:39:15.484970 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:39:15.484953 2573 scope.go:117] "RemoveContainer" containerID="a9f3b526bda7db9e06966858f70719a303bd9038a102ff40fd31dcbf6764b6a0" Aug 04 16:39:15.485252 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:39:15.485217 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9f3b526bda7db9e06966858f70719a303bd9038a102ff40fd31dcbf6764b6a0\": container with ID starting with a9f3b526bda7db9e06966858f70719a303bd9038a102ff40fd31dcbf6764b6a0 not found: ID does not exist" containerID="a9f3b526bda7db9e06966858f70719a303bd9038a102ff40fd31dcbf6764b6a0" Aug 04 16:39:15.485353 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:39:15.485261 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9f3b526bda7db9e06966858f70719a303bd9038a102ff40fd31dcbf6764b6a0"} err="failed to get container status \"a9f3b526bda7db9e06966858f70719a303bd9038a102ff40fd31dcbf6764b6a0\": rpc error: code = NotFound desc = could not find container \"a9f3b526bda7db9e06966858f70719a303bd9038a102ff40fd31dcbf6764b6a0\": container with ID starting with a9f3b526bda7db9e06966858f70719a303bd9038a102ff40fd31dcbf6764b6a0 not found: ID does not exist" Aug 04 16:39:15.497845 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:39:15.497816 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c2a82a29-794697655d-65hcq"] Aug 04 16:39:15.500768 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:39:15.500740 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c2a82a29-794697655d-65hcq"] Aug 04 16:39:17.079163 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:39:17.079128 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43913e0b-e433-4207-8df2-f651aa9491ce" path="/var/lib/kubelet/pods/43913e0b-e433-4207-8df2-f651aa9491ce/volumes" Aug 04 16:41:18.587086 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:18.587050 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-df4f93-7fbdf7f77d-5rm75"] Aug 04 16:41:18.587570 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:18.587506 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43913e0b-e433-4207-8df2-f651aa9491ce" containerName="maas-api" Aug 04 16:41:18.587570 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:18.587521 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="43913e0b-e433-4207-8df2-f651aa9491ce" containerName="maas-api" Aug 04 16:41:18.587667 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:18.587613 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="43913e0b-e433-4207-8df2-f651aa9491ce" containerName="maas-api" Aug 04 16:41:18.590784 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:18.590766 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df4f93-7fbdf7f77d-5rm75" Aug 04 16:41:18.593307 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:18.593282 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-df4f93-serving-cert\"" Aug 04 16:41:18.598642 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:18.598619 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-df4f93-7fbdf7f77d-5rm75"] Aug 04 16:41:18.672963 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:18.672926 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8604cba1-53a7-4ba5-9a74-2829b1225e51-maas-api-tls\") pod \"maas-api-e2e-inf-a-df4f93-7fbdf7f77d-5rm75\" (UID: \"8604cba1-53a7-4ba5-9a74-2829b1225e51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df4f93-7fbdf7f77d-5rm75" Aug 04 16:41:18.673133 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:18.672974 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8604cba1-53a7-4ba5-9a74-2829b1225e51-openshift-service-ca\") pod \"maas-api-e2e-inf-a-df4f93-7fbdf7f77d-5rm75\" (UID: \"8604cba1-53a7-4ba5-9a74-2829b1225e51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df4f93-7fbdf7f77d-5rm75" Aug 04 16:41:18.673133 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:18.672996 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rxtz\" (UniqueName: \"kubernetes.io/projected/8604cba1-53a7-4ba5-9a74-2829b1225e51-kube-api-access-4rxtz\") pod \"maas-api-e2e-inf-a-df4f93-7fbdf7f77d-5rm75\" (UID: \"8604cba1-53a7-4ba5-9a74-2829b1225e51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df4f93-7fbdf7f77d-5rm75" Aug 04 16:41:18.774334 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:18.774295 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8604cba1-53a7-4ba5-9a74-2829b1225e51-openshift-service-ca\") pod \"maas-api-e2e-inf-a-df4f93-7fbdf7f77d-5rm75\" (UID: \"8604cba1-53a7-4ba5-9a74-2829b1225e51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df4f93-7fbdf7f77d-5rm75" Aug 04 16:41:18.774334 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:18.774340 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rxtz\" (UniqueName: \"kubernetes.io/projected/8604cba1-53a7-4ba5-9a74-2829b1225e51-kube-api-access-4rxtz\") pod \"maas-api-e2e-inf-a-df4f93-7fbdf7f77d-5rm75\" (UID: \"8604cba1-53a7-4ba5-9a74-2829b1225e51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df4f93-7fbdf7f77d-5rm75" Aug 04 16:41:18.774586 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:18.774409 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8604cba1-53a7-4ba5-9a74-2829b1225e51-maas-api-tls\") pod \"maas-api-e2e-inf-a-df4f93-7fbdf7f77d-5rm75\" (UID: \"8604cba1-53a7-4ba5-9a74-2829b1225e51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df4f93-7fbdf7f77d-5rm75" Aug 04 16:41:18.775004 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:18.774982 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8604cba1-53a7-4ba5-9a74-2829b1225e51-openshift-service-ca\") pod \"maas-api-e2e-inf-a-df4f93-7fbdf7f77d-5rm75\" (UID: \"8604cba1-53a7-4ba5-9a74-2829b1225e51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df4f93-7fbdf7f77d-5rm75" Aug 04 16:41:18.777085 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:18.777056 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8604cba1-53a7-4ba5-9a74-2829b1225e51-maas-api-tls\") pod \"maas-api-e2e-inf-a-df4f93-7fbdf7f77d-5rm75\" (UID: \"8604cba1-53a7-4ba5-9a74-2829b1225e51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df4f93-7fbdf7f77d-5rm75" Aug 04 16:41:18.783034 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:18.783003 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rxtz\" (UniqueName: \"kubernetes.io/projected/8604cba1-53a7-4ba5-9a74-2829b1225e51-kube-api-access-4rxtz\") pod \"maas-api-e2e-inf-a-df4f93-7fbdf7f77d-5rm75\" (UID: \"8604cba1-53a7-4ba5-9a74-2829b1225e51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df4f93-7fbdf7f77d-5rm75" Aug 04 16:41:18.903537 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:18.903448 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df4f93-7fbdf7f77d-5rm75" Aug 04 16:41:19.045227 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:19.045196 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-df4f93-7fbdf7f77d-5rm75"] Aug 04 16:41:19.048406 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:41:19.048366 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8604cba1_53a7_4ba5_9a74_2829b1225e51.slice/crio-bff7be2cdccdf758f33f57665c1fd2a2e486afc94c7a88a8d53ec21d5d5d960e WatchSource:0}: Error finding container bff7be2cdccdf758f33f57665c1fd2a2e486afc94c7a88a8d53ec21d5d5d960e: Status 404 returned error can't find the container with id bff7be2cdccdf758f33f57665c1fd2a2e486afc94c7a88a8d53ec21d5d5d960e Aug 04 16:41:20.010401 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:20.010366 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df4f93-7fbdf7f77d-5rm75" event={"ID":"8604cba1-53a7-4ba5-9a74-2829b1225e51","Type":"ContainerStarted","Data":"c50b428e6e735e818bddbc8a56a66445c3857728e270c3b17388bf4aa37f6379"} Aug 04 16:41:20.010847 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:20.010416 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df4f93-7fbdf7f77d-5rm75" event={"ID":"8604cba1-53a7-4ba5-9a74-2829b1225e51","Type":"ContainerStarted","Data":"bff7be2cdccdf758f33f57665c1fd2a2e486afc94c7a88a8d53ec21d5d5d960e"} Aug 04 16:41:20.010847 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:20.010446 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df4f93-7fbdf7f77d-5rm75" Aug 04 16:41:20.029766 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:20.029713 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df4f93-7fbdf7f77d-5rm75" podStartSLOduration=1.5460553049999999 podStartE2EDuration="2.029692088s" totalImagesPullingTime="483.636783ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:41:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:41:20.029111319 +0000 UTC m=+2679.446653447" watchObservedRunningTime="2026-08-04 16:41:20.029692088 +0000 UTC m=+2679.447234217" Aug 04 16:41:26.020149 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:26.020120 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df4f93-7fbdf7f77d-5rm75" Aug 04 16:41:41.151306 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:41.151229 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-gvb4l_c16f82ab-8340-49ee-b41e-5685d4592fd2/console-operator/2.log" Aug 04 16:41:41.155439 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:41.155419 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mtbc5_11f9cbee-a773-420b-9873-5c8cb7679dd8/ovn-acl-logging/0.log" Aug 04 16:41:41.156807 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:41.156783 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-gvb4l_c16f82ab-8340-49ee-b41e-5685d4592fd2/console-operator/2.log" Aug 04 16:41:41.160786 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:41.160769 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mtbc5_11f9cbee-a773-420b-9873-5c8cb7679dd8/ovn-acl-logging/0.log" Aug 04 16:41:47.854677 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:47.854632 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1f0f0c-78bfcc586f-xkt7x"] Aug 04 16:41:47.859077 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:47.859048 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f0f0c-78bfcc586f-xkt7x" Aug 04 16:41:47.861369 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:47.861344 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-1f0f0c-serving-cert\"" Aug 04 16:41:47.868273 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:47.868243 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1f0f0c-78bfcc586f-xkt7x"] Aug 04 16:41:47.910960 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:47.910924 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s6q8\" (UniqueName: \"kubernetes.io/projected/8e8abbab-573e-4da3-9a04-8b44efebab53-kube-api-access-4s6q8\") pod \"maas-api-e2e-inf-b-1f0f0c-78bfcc586f-xkt7x\" (UID: \"8e8abbab-573e-4da3-9a04-8b44efebab53\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f0f0c-78bfcc586f-xkt7x" Aug 04 16:41:47.911131 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:47.910991 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e8abbab-573e-4da3-9a04-8b44efebab53-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1f0f0c-78bfcc586f-xkt7x\" (UID: \"8e8abbab-573e-4da3-9a04-8b44efebab53\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f0f0c-78bfcc586f-xkt7x" Aug 04 16:41:47.911131 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:47.911025 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e8abbab-573e-4da3-9a04-8b44efebab53-maas-api-tls\") pod \"maas-api-e2e-inf-b-1f0f0c-78bfcc586f-xkt7x\" (UID: \"8e8abbab-573e-4da3-9a04-8b44efebab53\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f0f0c-78bfcc586f-xkt7x" Aug 04 16:41:48.011480 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:48.011437 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s6q8\" (UniqueName: \"kubernetes.io/projected/8e8abbab-573e-4da3-9a04-8b44efebab53-kube-api-access-4s6q8\") pod \"maas-api-e2e-inf-b-1f0f0c-78bfcc586f-xkt7x\" (UID: \"8e8abbab-573e-4da3-9a04-8b44efebab53\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f0f0c-78bfcc586f-xkt7x" Aug 04 16:41:48.011719 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:48.011531 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e8abbab-573e-4da3-9a04-8b44efebab53-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1f0f0c-78bfcc586f-xkt7x\" (UID: \"8e8abbab-573e-4da3-9a04-8b44efebab53\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f0f0c-78bfcc586f-xkt7x" Aug 04 16:41:48.011719 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:48.011613 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e8abbab-573e-4da3-9a04-8b44efebab53-maas-api-tls\") pod \"maas-api-e2e-inf-b-1f0f0c-78bfcc586f-xkt7x\" (UID: \"8e8abbab-573e-4da3-9a04-8b44efebab53\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f0f0c-78bfcc586f-xkt7x" Aug 04 16:41:48.012191 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:48.012169 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e8abbab-573e-4da3-9a04-8b44efebab53-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1f0f0c-78bfcc586f-xkt7x\" (UID: \"8e8abbab-573e-4da3-9a04-8b44efebab53\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f0f0c-78bfcc586f-xkt7x" Aug 04 16:41:48.014245 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:48.014223 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e8abbab-573e-4da3-9a04-8b44efebab53-maas-api-tls\") pod \"maas-api-e2e-inf-b-1f0f0c-78bfcc586f-xkt7x\" (UID: \"8e8abbab-573e-4da3-9a04-8b44efebab53\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f0f0c-78bfcc586f-xkt7x" Aug 04 16:41:48.027863 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:48.023847 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s6q8\" (UniqueName: \"kubernetes.io/projected/8e8abbab-573e-4da3-9a04-8b44efebab53-kube-api-access-4s6q8\") pod \"maas-api-e2e-inf-b-1f0f0c-78bfcc586f-xkt7x\" (UID: \"8e8abbab-573e-4da3-9a04-8b44efebab53\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f0f0c-78bfcc586f-xkt7x" Aug 04 16:41:48.170020 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:48.169929 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f0f0c-78bfcc586f-xkt7x" Aug 04 16:41:48.312501 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:48.312471 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1f0f0c-78bfcc586f-xkt7x"] Aug 04 16:41:48.315103 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:41:48.315074 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e8abbab_573e_4da3_9a04_8b44efebab53.slice/crio-c4d2e203d1a370b6d119be3bb9351117ca725f894ab87f6fa5996cdd58366607 WatchSource:0}: Error finding container c4d2e203d1a370b6d119be3bb9351117ca725f894ab87f6fa5996cdd58366607: Status 404 returned error can't find the container with id c4d2e203d1a370b6d119be3bb9351117ca725f894ab87f6fa5996cdd58366607 Aug 04 16:41:49.130835 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:49.130798 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f0f0c-78bfcc586f-xkt7x" event={"ID":"8e8abbab-573e-4da3-9a04-8b44efebab53","Type":"ContainerStarted","Data":"ca4cafda74fd2eee0ee8904fe2a2c91001fe28ed029959a3f08a1974fc866d5a"} Aug 04 16:41:49.131285 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:49.130852 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f0f0c-78bfcc586f-xkt7x" event={"ID":"8e8abbab-573e-4da3-9a04-8b44efebab53","Type":"ContainerStarted","Data":"c4d2e203d1a370b6d119be3bb9351117ca725f894ab87f6fa5996cdd58366607"} Aug 04 16:41:49.131285 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:49.130914 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f0f0c-78bfcc586f-xkt7x" Aug 04 16:41:49.152776 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:49.152729 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f0f0c-78bfcc586f-xkt7x" podStartSLOduration=1.802195651 podStartE2EDuration="2.152713574s" totalImagesPullingTime="350.517923ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:41:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:41:49.150949469 +0000 UTC m=+2708.568491595" watchObservedRunningTime="2026-08-04 16:41:49.152713574 +0000 UTC m=+2708.570255702" Aug 04 16:41:55.140951 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:41:55.140922 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f0f0c-78bfcc586f-xkt7x" Aug 04 16:43:43.320358 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:43:43.320275 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-df4f93-7fbdf7f77d-5rm75"] Aug 04 16:43:43.320942 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:43:43.320517 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df4f93-7fbdf7f77d-5rm75" podUID="8604cba1-53a7-4ba5-9a74-2829b1225e51" containerName="maas-api" containerID="cri-o://c50b428e6e735e818bddbc8a56a66445c3857728e270c3b17388bf4aa37f6379" gracePeriod=30 Aug 04 16:43:43.581032 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:43:43.580969 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df4f93-7fbdf7f77d-5rm75" Aug 04 16:43:43.622762 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:43:43.622731 2573 generic.go:356] "Generic (PLEG): container finished" podID="8604cba1-53a7-4ba5-9a74-2829b1225e51" containerID="c50b428e6e735e818bddbc8a56a66445c3857728e270c3b17388bf4aa37f6379" exitCode=0 Aug 04 16:43:43.623010 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:43:43.622776 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df4f93-7fbdf7f77d-5rm75" event={"ID":"8604cba1-53a7-4ba5-9a74-2829b1225e51","Type":"ContainerDied","Data":"c50b428e6e735e818bddbc8a56a66445c3857728e270c3b17388bf4aa37f6379"} Aug 04 16:43:43.623010 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:43:43.622814 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df4f93-7fbdf7f77d-5rm75" event={"ID":"8604cba1-53a7-4ba5-9a74-2829b1225e51","Type":"ContainerDied","Data":"bff7be2cdccdf758f33f57665c1fd2a2e486afc94c7a88a8d53ec21d5d5d960e"} Aug 04 16:43:43.623010 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:43:43.622836 2573 scope.go:117] "RemoveContainer" containerID="c50b428e6e735e818bddbc8a56a66445c3857728e270c3b17388bf4aa37f6379" Aug 04 16:43:43.623010 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:43:43.622777 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df4f93-7fbdf7f77d-5rm75" Aug 04 16:43:43.631201 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:43:43.631178 2573 scope.go:117] "RemoveContainer" containerID="c50b428e6e735e818bddbc8a56a66445c3857728e270c3b17388bf4aa37f6379" Aug 04 16:43:43.631514 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:43:43.631492 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c50b428e6e735e818bddbc8a56a66445c3857728e270c3b17388bf4aa37f6379\": container with ID starting with c50b428e6e735e818bddbc8a56a66445c3857728e270c3b17388bf4aa37f6379 not found: ID does not exist" containerID="c50b428e6e735e818bddbc8a56a66445c3857728e270c3b17388bf4aa37f6379" Aug 04 16:43:43.631617 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:43:43.631522 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c50b428e6e735e818bddbc8a56a66445c3857728e270c3b17388bf4aa37f6379"} err="failed to get container status \"c50b428e6e735e818bddbc8a56a66445c3857728e270c3b17388bf4aa37f6379\": rpc error: code = NotFound desc = could not find container \"c50b428e6e735e818bddbc8a56a66445c3857728e270c3b17388bf4aa37f6379\": container with ID starting with c50b428e6e735e818bddbc8a56a66445c3857728e270c3b17388bf4aa37f6379 not found: ID does not exist" Aug 04 16:43:43.709116 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:43:43.709078 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rxtz\" (UniqueName: \"kubernetes.io/projected/8604cba1-53a7-4ba5-9a74-2829b1225e51-kube-api-access-4rxtz\") pod \"8604cba1-53a7-4ba5-9a74-2829b1225e51\" (UID: \"8604cba1-53a7-4ba5-9a74-2829b1225e51\") " Aug 04 16:43:43.709328 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:43:43.709164 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8604cba1-53a7-4ba5-9a74-2829b1225e51-maas-api-tls\") pod \"8604cba1-53a7-4ba5-9a74-2829b1225e51\" (UID: \"8604cba1-53a7-4ba5-9a74-2829b1225e51\") " Aug 04 16:43:43.709328 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:43:43.709225 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8604cba1-53a7-4ba5-9a74-2829b1225e51-openshift-service-ca\") pod \"8604cba1-53a7-4ba5-9a74-2829b1225e51\" (UID: \"8604cba1-53a7-4ba5-9a74-2829b1225e51\") " Aug 04 16:43:43.709595 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:43:43.709562 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8604cba1-53a7-4ba5-9a74-2829b1225e51-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8604cba1-53a7-4ba5-9a74-2829b1225e51" (UID: "8604cba1-53a7-4ba5-9a74-2829b1225e51"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:43:43.711367 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:43:43.711332 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8604cba1-53a7-4ba5-9a74-2829b1225e51-kube-api-access-4rxtz" (OuterVolumeSpecName: "kube-api-access-4rxtz") pod "8604cba1-53a7-4ba5-9a74-2829b1225e51" (UID: "8604cba1-53a7-4ba5-9a74-2829b1225e51"). InnerVolumeSpecName "kube-api-access-4rxtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:43:43.711568 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:43:43.711512 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8604cba1-53a7-4ba5-9a74-2829b1225e51-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8604cba1-53a7-4ba5-9a74-2829b1225e51" (UID: "8604cba1-53a7-4ba5-9a74-2829b1225e51"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:43:43.810810 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:43:43.810779 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rxtz\" (UniqueName: \"kubernetes.io/projected/8604cba1-53a7-4ba5-9a74-2829b1225e51-kube-api-access-4rxtz\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:43:43.810810 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:43:43.810805 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8604cba1-53a7-4ba5-9a74-2829b1225e51-maas-api-tls\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:43:43.810810 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:43:43.810814 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8604cba1-53a7-4ba5-9a74-2829b1225e51-openshift-service-ca\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:43:43.946607 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:43:43.946573 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-df4f93-7fbdf7f77d-5rm75"] Aug 04 16:43:43.951309 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:43:43.951285 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-df4f93-7fbdf7f77d-5rm75"] Aug 04 16:43:45.077928 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:43:45.077890 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8604cba1-53a7-4ba5-9a74-2829b1225e51" path="/var/lib/kubelet/pods/8604cba1-53a7-4ba5-9a74-2829b1225e51/volumes" Aug 04 16:44:00.972086 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:00.972054 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1f0f0c-78bfcc586f-xkt7x"] Aug 04 16:44:00.972474 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:00.972254 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f0f0c-78bfcc586f-xkt7x" podUID="8e8abbab-573e-4da3-9a04-8b44efebab53" containerName="maas-api" containerID="cri-o://ca4cafda74fd2eee0ee8904fe2a2c91001fe28ed029959a3f08a1974fc866d5a" gracePeriod=30 Aug 04 16:44:01.223120 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:01.223054 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f0f0c-78bfcc586f-xkt7x" Aug 04 16:44:01.248799 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:01.248771 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e8abbab-573e-4da3-9a04-8b44efebab53-maas-api-tls\") pod \"8e8abbab-573e-4da3-9a04-8b44efebab53\" (UID: \"8e8abbab-573e-4da3-9a04-8b44efebab53\") " Aug 04 16:44:01.248949 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:01.248809 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4s6q8\" (UniqueName: \"kubernetes.io/projected/8e8abbab-573e-4da3-9a04-8b44efebab53-kube-api-access-4s6q8\") pod \"8e8abbab-573e-4da3-9a04-8b44efebab53\" (UID: \"8e8abbab-573e-4da3-9a04-8b44efebab53\") " Aug 04 16:44:01.248949 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:01.248864 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e8abbab-573e-4da3-9a04-8b44efebab53-openshift-service-ca\") pod \"8e8abbab-573e-4da3-9a04-8b44efebab53\" (UID: \"8e8abbab-573e-4da3-9a04-8b44efebab53\") " Aug 04 16:44:01.249280 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:01.249252 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e8abbab-573e-4da3-9a04-8b44efebab53-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8e8abbab-573e-4da3-9a04-8b44efebab53" (UID: "8e8abbab-573e-4da3-9a04-8b44efebab53"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:44:01.250889 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:01.250866 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e8abbab-573e-4da3-9a04-8b44efebab53-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8e8abbab-573e-4da3-9a04-8b44efebab53" (UID: "8e8abbab-573e-4da3-9a04-8b44efebab53"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:44:01.251009 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:01.250984 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e8abbab-573e-4da3-9a04-8b44efebab53-kube-api-access-4s6q8" (OuterVolumeSpecName: "kube-api-access-4s6q8") pod "8e8abbab-573e-4da3-9a04-8b44efebab53" (UID: "8e8abbab-573e-4da3-9a04-8b44efebab53"). InnerVolumeSpecName "kube-api-access-4s6q8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:44:01.349497 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:01.349456 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e8abbab-573e-4da3-9a04-8b44efebab53-openshift-service-ca\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:44:01.349497 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:01.349490 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e8abbab-573e-4da3-9a04-8b44efebab53-maas-api-tls\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:44:01.349497 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:01.349503 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s6q8\" (UniqueName: \"kubernetes.io/projected/8e8abbab-573e-4da3-9a04-8b44efebab53-kube-api-access-4s6q8\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:44:01.698784 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:01.698757 2573 generic.go:356] "Generic (PLEG): container finished" podID="8e8abbab-573e-4da3-9a04-8b44efebab53" containerID="ca4cafda74fd2eee0ee8904fe2a2c91001fe28ed029959a3f08a1974fc866d5a" exitCode=0 Aug 04 16:44:01.698964 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:01.698802 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f0f0c-78bfcc586f-xkt7x" Aug 04 16:44:01.698964 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:01.698843 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f0f0c-78bfcc586f-xkt7x" event={"ID":"8e8abbab-573e-4da3-9a04-8b44efebab53","Type":"ContainerDied","Data":"ca4cafda74fd2eee0ee8904fe2a2c91001fe28ed029959a3f08a1974fc866d5a"} Aug 04 16:44:01.698964 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:01.698903 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f0f0c-78bfcc586f-xkt7x" event={"ID":"8e8abbab-573e-4da3-9a04-8b44efebab53","Type":"ContainerDied","Data":"c4d2e203d1a370b6d119be3bb9351117ca725f894ab87f6fa5996cdd58366607"} Aug 04 16:44:01.698964 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:01.698925 2573 scope.go:117] "RemoveContainer" containerID="ca4cafda74fd2eee0ee8904fe2a2c91001fe28ed029959a3f08a1974fc866d5a" Aug 04 16:44:01.707147 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:01.707129 2573 scope.go:117] "RemoveContainer" containerID="ca4cafda74fd2eee0ee8904fe2a2c91001fe28ed029959a3f08a1974fc866d5a" Aug 04 16:44:01.707412 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:44:01.707394 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca4cafda74fd2eee0ee8904fe2a2c91001fe28ed029959a3f08a1974fc866d5a\": container with ID starting with ca4cafda74fd2eee0ee8904fe2a2c91001fe28ed029959a3f08a1974fc866d5a not found: ID does not exist" containerID="ca4cafda74fd2eee0ee8904fe2a2c91001fe28ed029959a3f08a1974fc866d5a" Aug 04 16:44:01.707457 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:01.707421 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca4cafda74fd2eee0ee8904fe2a2c91001fe28ed029959a3f08a1974fc866d5a"} err="failed to get container status \"ca4cafda74fd2eee0ee8904fe2a2c91001fe28ed029959a3f08a1974fc866d5a\": rpc error: code = NotFound desc = could not find container \"ca4cafda74fd2eee0ee8904fe2a2c91001fe28ed029959a3f08a1974fc866d5a\": container with ID starting with ca4cafda74fd2eee0ee8904fe2a2c91001fe28ed029959a3f08a1974fc866d5a not found: ID does not exist" Aug 04 16:44:01.718986 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:01.718946 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1f0f0c-78bfcc586f-xkt7x"] Aug 04 16:44:01.720618 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:01.720599 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1f0f0c-78bfcc586f-xkt7x"] Aug 04 16:44:03.078658 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:03.078626 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e8abbab-573e-4da3-9a04-8b44efebab53" path="/var/lib/kubelet/pods/8e8abbab-573e-4da3-9a04-8b44efebab53/volumes" Aug 04 16:44:12.744403 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:12.744369 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-78e391-57457968b6-dds4t"] Aug 04 16:44:12.744896 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:12.744859 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e8abbab-573e-4da3-9a04-8b44efebab53" containerName="maas-api" Aug 04 16:44:12.744896 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:12.744879 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e8abbab-573e-4da3-9a04-8b44efebab53" containerName="maas-api" Aug 04 16:44:12.745010 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:12.744919 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8604cba1-53a7-4ba5-9a74-2829b1225e51" containerName="maas-api" Aug 04 16:44:12.745010 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:12.744929 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8604cba1-53a7-4ba5-9a74-2829b1225e51" containerName="maas-api" Aug 04 16:44:12.745108 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:12.745013 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e8abbab-573e-4da3-9a04-8b44efebab53" containerName="maas-api" Aug 04 16:44:12.745108 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:12.745030 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8604cba1-53a7-4ba5-9a74-2829b1225e51" containerName="maas-api" Aug 04 16:44:12.750626 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:12.749521 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78e391-57457968b6-dds4t" Aug 04 16:44:12.752192 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:12.752161 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-78e391-serving-cert\"" Aug 04 16:44:12.764856 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:12.764823 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-78e391-57457968b6-dds4t"] Aug 04 16:44:12.850856 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:12.850817 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0711bc29-550e-45d5-8294-752ede5863d7-maas-api-tls\") pod \"maas-api-e2e-api-a-78e391-57457968b6-dds4t\" (UID: \"0711bc29-550e-45d5-8294-752ede5863d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78e391-57457968b6-dds4t" Aug 04 16:44:12.851038 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:12.850880 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0711bc29-550e-45d5-8294-752ede5863d7-openshift-service-ca\") pod \"maas-api-e2e-api-a-78e391-57457968b6-dds4t\" (UID: \"0711bc29-550e-45d5-8294-752ede5863d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78e391-57457968b6-dds4t" Aug 04 16:44:12.851038 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:12.850927 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qwg6\" (UniqueName: \"kubernetes.io/projected/0711bc29-550e-45d5-8294-752ede5863d7-kube-api-access-8qwg6\") pod \"maas-api-e2e-api-a-78e391-57457968b6-dds4t\" (UID: \"0711bc29-550e-45d5-8294-752ede5863d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78e391-57457968b6-dds4t" Aug 04 16:44:12.952364 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:12.952325 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0711bc29-550e-45d5-8294-752ede5863d7-maas-api-tls\") pod \"maas-api-e2e-api-a-78e391-57457968b6-dds4t\" (UID: \"0711bc29-550e-45d5-8294-752ede5863d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78e391-57457968b6-dds4t" Aug 04 16:44:12.952568 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:12.952393 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0711bc29-550e-45d5-8294-752ede5863d7-openshift-service-ca\") pod \"maas-api-e2e-api-a-78e391-57457968b6-dds4t\" (UID: \"0711bc29-550e-45d5-8294-752ede5863d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78e391-57457968b6-dds4t" Aug 04 16:44:12.952568 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:12.952419 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qwg6\" (UniqueName: \"kubernetes.io/projected/0711bc29-550e-45d5-8294-752ede5863d7-kube-api-access-8qwg6\") pod \"maas-api-e2e-api-a-78e391-57457968b6-dds4t\" (UID: \"0711bc29-550e-45d5-8294-752ede5863d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78e391-57457968b6-dds4t" Aug 04 16:44:12.953204 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:12.953166 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0711bc29-550e-45d5-8294-752ede5863d7-openshift-service-ca\") pod \"maas-api-e2e-api-a-78e391-57457968b6-dds4t\" (UID: \"0711bc29-550e-45d5-8294-752ede5863d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78e391-57457968b6-dds4t" Aug 04 16:44:12.955160 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:12.955136 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0711bc29-550e-45d5-8294-752ede5863d7-maas-api-tls\") pod \"maas-api-e2e-api-a-78e391-57457968b6-dds4t\" (UID: \"0711bc29-550e-45d5-8294-752ede5863d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78e391-57457968b6-dds4t" Aug 04 16:44:12.960361 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:12.960340 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qwg6\" (UniqueName: \"kubernetes.io/projected/0711bc29-550e-45d5-8294-752ede5863d7-kube-api-access-8qwg6\") pod \"maas-api-e2e-api-a-78e391-57457968b6-dds4t\" (UID: \"0711bc29-550e-45d5-8294-752ede5863d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78e391-57457968b6-dds4t" Aug 04 16:44:13.063647 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:13.063534 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78e391-57457968b6-dds4t" Aug 04 16:44:13.211891 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:13.211860 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-78e391-57457968b6-dds4t"] Aug 04 16:44:13.214606 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:44:13.214581 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0711bc29_550e_45d5_8294_752ede5863d7.slice/crio-e7c33a0893a538356878b53c27f80351237af72f189c169fa8461ad3bdd927e4 WatchSource:0}: Error finding container e7c33a0893a538356878b53c27f80351237af72f189c169fa8461ad3bdd927e4: Status 404 returned error can't find the container with id e7c33a0893a538356878b53c27f80351237af72f189c169fa8461ad3bdd927e4 Aug 04 16:44:13.215867 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:13.215848 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 16:44:13.752563 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:13.752516 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78e391-57457968b6-dds4t" event={"ID":"0711bc29-550e-45d5-8294-752ede5863d7","Type":"ContainerStarted","Data":"e7c33a0893a538356878b53c27f80351237af72f189c169fa8461ad3bdd927e4"} Aug 04 16:44:14.759740 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:14.759707 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78e391-57457968b6-dds4t" event={"ID":"0711bc29-550e-45d5-8294-752ede5863d7","Type":"ContainerStarted","Data":"5c11b50467eefe959c4e6162c8ad27926b03f5666943d9e1ad532c692874a6be"} Aug 04 16:44:14.760241 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:14.759817 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78e391-57457968b6-dds4t" Aug 04 16:44:14.778769 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:14.778714 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78e391-57457968b6-dds4t" podStartSLOduration=2.236539362 podStartE2EDuration="2.778697467s" totalImagesPullingTime="542.158105ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:44:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:44:14.777887753 +0000 UTC m=+2854.195429879" watchObservedRunningTime="2026-08-04 16:44:14.778697467 +0000 UTC m=+2854.196239592" Aug 04 16:44:20.770227 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:20.770198 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78e391-57457968b6-dds4t" Aug 04 16:44:42.442115 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:42.442083 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e13060-767bfc8dc-69rxb"] Aug 04 16:44:42.446635 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:42.446613 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e13060-767bfc8dc-69rxb" Aug 04 16:44:42.449053 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:42.449031 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-e13060-serving-cert\"" Aug 04 16:44:42.456803 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:42.456776 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e13060-767bfc8dc-69rxb"] Aug 04 16:44:42.498166 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:42.498132 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmgcp\" (UniqueName: \"kubernetes.io/projected/60644016-acf8-44f5-8c2d-21d85fd9fc4a-kube-api-access-tmgcp\") pod \"maas-api-e2e-api-b-e13060-767bfc8dc-69rxb\" (UID: \"60644016-acf8-44f5-8c2d-21d85fd9fc4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e13060-767bfc8dc-69rxb" Aug 04 16:44:42.498166 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:42.498173 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60644016-acf8-44f5-8c2d-21d85fd9fc4a-openshift-service-ca\") pod \"maas-api-e2e-api-b-e13060-767bfc8dc-69rxb\" (UID: \"60644016-acf8-44f5-8c2d-21d85fd9fc4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e13060-767bfc8dc-69rxb" Aug 04 16:44:42.498394 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:42.498190 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60644016-acf8-44f5-8c2d-21d85fd9fc4a-maas-api-tls\") pod \"maas-api-e2e-api-b-e13060-767bfc8dc-69rxb\" (UID: \"60644016-acf8-44f5-8c2d-21d85fd9fc4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e13060-767bfc8dc-69rxb" Aug 04 16:44:42.599025 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:42.598983 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmgcp\" (UniqueName: \"kubernetes.io/projected/60644016-acf8-44f5-8c2d-21d85fd9fc4a-kube-api-access-tmgcp\") pod \"maas-api-e2e-api-b-e13060-767bfc8dc-69rxb\" (UID: \"60644016-acf8-44f5-8c2d-21d85fd9fc4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e13060-767bfc8dc-69rxb" Aug 04 16:44:42.599025 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:42.599026 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60644016-acf8-44f5-8c2d-21d85fd9fc4a-openshift-service-ca\") pod \"maas-api-e2e-api-b-e13060-767bfc8dc-69rxb\" (UID: \"60644016-acf8-44f5-8c2d-21d85fd9fc4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e13060-767bfc8dc-69rxb" Aug 04 16:44:42.599281 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:42.599048 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60644016-acf8-44f5-8c2d-21d85fd9fc4a-maas-api-tls\") pod \"maas-api-e2e-api-b-e13060-767bfc8dc-69rxb\" (UID: \"60644016-acf8-44f5-8c2d-21d85fd9fc4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e13060-767bfc8dc-69rxb" Aug 04 16:44:42.599281 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:44:42.599163 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-e13060-serving-cert: secret "maas-api-e2e-api-b-e13060-serving-cert" not found Aug 04 16:44:42.599281 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:44:42.599215 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60644016-acf8-44f5-8c2d-21d85fd9fc4a-maas-api-tls podName:60644016-acf8-44f5-8c2d-21d85fd9fc4a nodeName:}" failed. No retries permitted until 2026-08-04 16:44:43.099199739 +0000 UTC m=+2882.516741843 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/60644016-acf8-44f5-8c2d-21d85fd9fc4a-maas-api-tls") pod "maas-api-e2e-api-b-e13060-767bfc8dc-69rxb" (UID: "60644016-acf8-44f5-8c2d-21d85fd9fc4a") : secret "maas-api-e2e-api-b-e13060-serving-cert" not found Aug 04 16:44:42.599738 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:42.599717 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60644016-acf8-44f5-8c2d-21d85fd9fc4a-openshift-service-ca\") pod \"maas-api-e2e-api-b-e13060-767bfc8dc-69rxb\" (UID: \"60644016-acf8-44f5-8c2d-21d85fd9fc4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e13060-767bfc8dc-69rxb" Aug 04 16:44:42.608152 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:42.608124 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmgcp\" (UniqueName: \"kubernetes.io/projected/60644016-acf8-44f5-8c2d-21d85fd9fc4a-kube-api-access-tmgcp\") pod \"maas-api-e2e-api-b-e13060-767bfc8dc-69rxb\" (UID: \"60644016-acf8-44f5-8c2d-21d85fd9fc4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e13060-767bfc8dc-69rxb" Aug 04 16:44:43.102873 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:43.102828 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60644016-acf8-44f5-8c2d-21d85fd9fc4a-maas-api-tls\") pod \"maas-api-e2e-api-b-e13060-767bfc8dc-69rxb\" (UID: \"60644016-acf8-44f5-8c2d-21d85fd9fc4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e13060-767bfc8dc-69rxb" Aug 04 16:44:43.105312 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:43.105286 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60644016-acf8-44f5-8c2d-21d85fd9fc4a-maas-api-tls\") pod \"maas-api-e2e-api-b-e13060-767bfc8dc-69rxb\" (UID: \"60644016-acf8-44f5-8c2d-21d85fd9fc4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e13060-767bfc8dc-69rxb" Aug 04 16:44:43.359174 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:43.359098 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e13060-767bfc8dc-69rxb" Aug 04 16:44:43.493033 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:43.492853 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e13060-767bfc8dc-69rxb"] Aug 04 16:44:43.495942 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:44:43.495903 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60644016_acf8_44f5_8c2d_21d85fd9fc4a.slice/crio-d7259c9fbb8fee731e5c8ff72b3f53786ba4c31bb30893b9a07457b065ff5a5f WatchSource:0}: Error finding container d7259c9fbb8fee731e5c8ff72b3f53786ba4c31bb30893b9a07457b065ff5a5f: Status 404 returned error can't find the container with id d7259c9fbb8fee731e5c8ff72b3f53786ba4c31bb30893b9a07457b065ff5a5f Aug 04 16:44:43.891650 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:43.891608 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e13060-767bfc8dc-69rxb" event={"ID":"60644016-acf8-44f5-8c2d-21d85fd9fc4a","Type":"ContainerStarted","Data":"d7259c9fbb8fee731e5c8ff72b3f53786ba4c31bb30893b9a07457b065ff5a5f"} Aug 04 16:44:44.897358 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:44.897325 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e13060-767bfc8dc-69rxb" event={"ID":"60644016-acf8-44f5-8c2d-21d85fd9fc4a","Type":"ContainerStarted","Data":"bc93454c9a0321cfdcc7ec3a6421e543ab614cb3c94c0c440867e72c46d70faa"} Aug 04 16:44:44.897905 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:44.897435 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e13060-767bfc8dc-69rxb" Aug 04 16:44:44.914491 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:44.914440 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e13060-767bfc8dc-69rxb" podStartSLOduration=2.410436546 podStartE2EDuration="2.91442231s" totalImagesPullingTime="503.985764ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:44:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:44:44.912995327 +0000 UTC m=+2884.330537477" watchObservedRunningTime="2026-08-04 16:44:44.91442231 +0000 UTC m=+2884.331964436" Aug 04 16:44:50.906901 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:44:50.906874 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e13060-767bfc8dc-69rxb" Aug 04 16:45:11.655964 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:11.655927 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-78e391-57457968b6-dds4t"] Aug 04 16:45:11.656437 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:11.656192 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78e391-57457968b6-dds4t" podUID="0711bc29-550e-45d5-8294-752ede5863d7" containerName="maas-api" containerID="cri-o://5c11b50467eefe959c4e6162c8ad27926b03f5666943d9e1ad532c692874a6be" gracePeriod=30 Aug 04 16:45:11.894911 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:11.894883 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78e391-57457968b6-dds4t" Aug 04 16:45:11.952005 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:11.951979 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0711bc29-550e-45d5-8294-752ede5863d7-maas-api-tls\") pod \"0711bc29-550e-45d5-8294-752ede5863d7\" (UID: \"0711bc29-550e-45d5-8294-752ede5863d7\") " Aug 04 16:45:11.952161 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:11.952072 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0711bc29-550e-45d5-8294-752ede5863d7-openshift-service-ca\") pod \"0711bc29-550e-45d5-8294-752ede5863d7\" (UID: \"0711bc29-550e-45d5-8294-752ede5863d7\") " Aug 04 16:45:11.952161 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:11.952105 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qwg6\" (UniqueName: \"kubernetes.io/projected/0711bc29-550e-45d5-8294-752ede5863d7-kube-api-access-8qwg6\") pod \"0711bc29-550e-45d5-8294-752ede5863d7\" (UID: \"0711bc29-550e-45d5-8294-752ede5863d7\") " Aug 04 16:45:11.952446 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:11.952425 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0711bc29-550e-45d5-8294-752ede5863d7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0711bc29-550e-45d5-8294-752ede5863d7" (UID: "0711bc29-550e-45d5-8294-752ede5863d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:45:11.954084 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:11.954048 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0711bc29-550e-45d5-8294-752ede5863d7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0711bc29-550e-45d5-8294-752ede5863d7" (UID: "0711bc29-550e-45d5-8294-752ede5863d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:45:11.954231 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:11.954162 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0711bc29-550e-45d5-8294-752ede5863d7-kube-api-access-8qwg6" (OuterVolumeSpecName: "kube-api-access-8qwg6") pod "0711bc29-550e-45d5-8294-752ede5863d7" (UID: "0711bc29-550e-45d5-8294-752ede5863d7"). InnerVolumeSpecName "kube-api-access-8qwg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:45:12.013595 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:12.013547 2573 generic.go:356] "Generic (PLEG): container finished" podID="0711bc29-550e-45d5-8294-752ede5863d7" containerID="5c11b50467eefe959c4e6162c8ad27926b03f5666943d9e1ad532c692874a6be" exitCode=0 Aug 04 16:45:12.013751 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:12.013606 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78e391-57457968b6-dds4t" Aug 04 16:45:12.013751 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:12.013599 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78e391-57457968b6-dds4t" event={"ID":"0711bc29-550e-45d5-8294-752ede5863d7","Type":"ContainerDied","Data":"5c11b50467eefe959c4e6162c8ad27926b03f5666943d9e1ad532c692874a6be"} Aug 04 16:45:12.013751 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:12.013664 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78e391-57457968b6-dds4t" event={"ID":"0711bc29-550e-45d5-8294-752ede5863d7","Type":"ContainerDied","Data":"e7c33a0893a538356878b53c27f80351237af72f189c169fa8461ad3bdd927e4"} Aug 04 16:45:12.013751 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:12.013686 2573 scope.go:117] "RemoveContainer" containerID="5c11b50467eefe959c4e6162c8ad27926b03f5666943d9e1ad532c692874a6be" Aug 04 16:45:12.022203 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:12.022187 2573 scope.go:117] "RemoveContainer" containerID="5c11b50467eefe959c4e6162c8ad27926b03f5666943d9e1ad532c692874a6be" Aug 04 16:45:12.022486 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:45:12.022465 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c11b50467eefe959c4e6162c8ad27926b03f5666943d9e1ad532c692874a6be\": container with ID starting with 5c11b50467eefe959c4e6162c8ad27926b03f5666943d9e1ad532c692874a6be not found: ID does not exist" containerID="5c11b50467eefe959c4e6162c8ad27926b03f5666943d9e1ad532c692874a6be" Aug 04 16:45:12.022525 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:12.022496 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c11b50467eefe959c4e6162c8ad27926b03f5666943d9e1ad532c692874a6be"} err="failed to get container status \"5c11b50467eefe959c4e6162c8ad27926b03f5666943d9e1ad532c692874a6be\": rpc error: code = NotFound desc = could not find container \"5c11b50467eefe959c4e6162c8ad27926b03f5666943d9e1ad532c692874a6be\": container with ID starting with 5c11b50467eefe959c4e6162c8ad27926b03f5666943d9e1ad532c692874a6be not found: ID does not exist" Aug 04 16:45:12.053039 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:12.053006 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0711bc29-550e-45d5-8294-752ede5863d7-openshift-service-ca\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:45:12.053204 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:12.053033 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qwg6\" (UniqueName: \"kubernetes.io/projected/0711bc29-550e-45d5-8294-752ede5863d7-kube-api-access-8qwg6\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:45:12.053204 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:12.053076 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0711bc29-550e-45d5-8294-752ede5863d7-maas-api-tls\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:45:12.054899 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:12.054872 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-78e391-57457968b6-dds4t"] Aug 04 16:45:12.057814 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:12.057791 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-78e391-57457968b6-dds4t"] Aug 04 16:45:13.083243 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:13.079954 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0711bc29-550e-45d5-8294-752ede5863d7" path="/var/lib/kubelet/pods/0711bc29-550e-45d5-8294-752ede5863d7/volumes" Aug 04 16:45:30.352517 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:30.352476 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e13060-767bfc8dc-69rxb"] Aug 04 16:45:30.353112 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:30.352756 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e13060-767bfc8dc-69rxb" podUID="60644016-acf8-44f5-8c2d-21d85fd9fc4a" containerName="maas-api" containerID="cri-o://bc93454c9a0321cfdcc7ec3a6421e543ab614cb3c94c0c440867e72c46d70faa" gracePeriod=30 Aug 04 16:45:30.592959 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:30.592932 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e13060-767bfc8dc-69rxb" Aug 04 16:45:30.715635 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:30.715605 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmgcp\" (UniqueName: \"kubernetes.io/projected/60644016-acf8-44f5-8c2d-21d85fd9fc4a-kube-api-access-tmgcp\") pod \"60644016-acf8-44f5-8c2d-21d85fd9fc4a\" (UID: \"60644016-acf8-44f5-8c2d-21d85fd9fc4a\") " Aug 04 16:45:30.715793 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:30.715670 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60644016-acf8-44f5-8c2d-21d85fd9fc4a-maas-api-tls\") pod \"60644016-acf8-44f5-8c2d-21d85fd9fc4a\" (UID: \"60644016-acf8-44f5-8c2d-21d85fd9fc4a\") " Aug 04 16:45:30.715793 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:30.715723 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60644016-acf8-44f5-8c2d-21d85fd9fc4a-openshift-service-ca\") pod \"60644016-acf8-44f5-8c2d-21d85fd9fc4a\" (UID: \"60644016-acf8-44f5-8c2d-21d85fd9fc4a\") " Aug 04 16:45:30.716145 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:30.716121 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60644016-acf8-44f5-8c2d-21d85fd9fc4a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "60644016-acf8-44f5-8c2d-21d85fd9fc4a" (UID: "60644016-acf8-44f5-8c2d-21d85fd9fc4a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:45:30.717814 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:30.717790 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60644016-acf8-44f5-8c2d-21d85fd9fc4a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "60644016-acf8-44f5-8c2d-21d85fd9fc4a" (UID: "60644016-acf8-44f5-8c2d-21d85fd9fc4a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:45:30.717914 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:30.717816 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60644016-acf8-44f5-8c2d-21d85fd9fc4a-kube-api-access-tmgcp" (OuterVolumeSpecName: "kube-api-access-tmgcp") pod "60644016-acf8-44f5-8c2d-21d85fd9fc4a" (UID: "60644016-acf8-44f5-8c2d-21d85fd9fc4a"). InnerVolumeSpecName "kube-api-access-tmgcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:45:30.817300 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:30.817264 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmgcp\" (UniqueName: \"kubernetes.io/projected/60644016-acf8-44f5-8c2d-21d85fd9fc4a-kube-api-access-tmgcp\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:45:30.817300 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:30.817293 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60644016-acf8-44f5-8c2d-21d85fd9fc4a-maas-api-tls\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:45:30.817300 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:30.817303 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60644016-acf8-44f5-8c2d-21d85fd9fc4a-openshift-service-ca\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:45:31.094000 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:31.093972 2573 generic.go:356] "Generic (PLEG): container finished" podID="60644016-acf8-44f5-8c2d-21d85fd9fc4a" containerID="bc93454c9a0321cfdcc7ec3a6421e543ab614cb3c94c0c440867e72c46d70faa" exitCode=0 Aug 04 16:45:31.094306 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:31.094016 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e13060-767bfc8dc-69rxb" Aug 04 16:45:31.094306 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:31.094031 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e13060-767bfc8dc-69rxb" event={"ID":"60644016-acf8-44f5-8c2d-21d85fd9fc4a","Type":"ContainerDied","Data":"bc93454c9a0321cfdcc7ec3a6421e543ab614cb3c94c0c440867e72c46d70faa"} Aug 04 16:45:31.094306 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:31.094072 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e13060-767bfc8dc-69rxb" event={"ID":"60644016-acf8-44f5-8c2d-21d85fd9fc4a","Type":"ContainerDied","Data":"d7259c9fbb8fee731e5c8ff72b3f53786ba4c31bb30893b9a07457b065ff5a5f"} Aug 04 16:45:31.094306 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:31.094094 2573 scope.go:117] "RemoveContainer" containerID="bc93454c9a0321cfdcc7ec3a6421e543ab614cb3c94c0c440867e72c46d70faa" Aug 04 16:45:31.102253 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:31.102235 2573 scope.go:117] "RemoveContainer" containerID="bc93454c9a0321cfdcc7ec3a6421e543ab614cb3c94c0c440867e72c46d70faa" Aug 04 16:45:31.102516 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:45:31.102497 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc93454c9a0321cfdcc7ec3a6421e543ab614cb3c94c0c440867e72c46d70faa\": container with ID starting with bc93454c9a0321cfdcc7ec3a6421e543ab614cb3c94c0c440867e72c46d70faa not found: ID does not exist" containerID="bc93454c9a0321cfdcc7ec3a6421e543ab614cb3c94c0c440867e72c46d70faa" Aug 04 16:45:31.102587 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:31.102524 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc93454c9a0321cfdcc7ec3a6421e543ab614cb3c94c0c440867e72c46d70faa"} err="failed to get container status \"bc93454c9a0321cfdcc7ec3a6421e543ab614cb3c94c0c440867e72c46d70faa\": rpc error: code = NotFound desc = could not find container \"bc93454c9a0321cfdcc7ec3a6421e543ab614cb3c94c0c440867e72c46d70faa\": container with ID starting with bc93454c9a0321cfdcc7ec3a6421e543ab614cb3c94c0c440867e72c46d70faa not found: ID does not exist" Aug 04 16:45:31.109545 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:31.109521 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e13060-767bfc8dc-69rxb"] Aug 04 16:45:31.113299 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:31.113280 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e13060-767bfc8dc-69rxb"] Aug 04 16:45:33.078023 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:33.077990 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60644016-acf8-44f5-8c2d-21d85fd9fc4a" path="/var/lib/kubelet/pods/60644016-acf8-44f5-8c2d-21d85fd9fc4a/volumes" Aug 04 16:45:42.117852 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:42.117820 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-90d707-5df6477b5-rbbnn"] Aug 04 16:45:42.118215 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:42.118121 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0711bc29-550e-45d5-8294-752ede5863d7" containerName="maas-api" Aug 04 16:45:42.118215 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:42.118132 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0711bc29-550e-45d5-8294-752ede5863d7" containerName="maas-api" Aug 04 16:45:42.118215 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:42.118142 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60644016-acf8-44f5-8c2d-21d85fd9fc4a" containerName="maas-api" Aug 04 16:45:42.118215 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:42.118147 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="60644016-acf8-44f5-8c2d-21d85fd9fc4a" containerName="maas-api" Aug 04 16:45:42.118344 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:42.118221 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0711bc29-550e-45d5-8294-752ede5863d7" containerName="maas-api" Aug 04 16:45:42.118344 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:42.118232 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="60644016-acf8-44f5-8c2d-21d85fd9fc4a" containerName="maas-api" Aug 04 16:45:42.122691 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:42.122655 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90d707-5df6477b5-rbbnn" Aug 04 16:45:42.124686 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:42.124662 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-90d707-serving-cert\"" Aug 04 16:45:42.133012 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:42.132967 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-90d707-5df6477b5-rbbnn"] Aug 04 16:45:42.212931 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:42.212893 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67a2f33c-cef6-482f-b9bb-4c39342b7e94-maas-api-tls\") pod \"maas-api-e2e-shared-a-90d707-5df6477b5-rbbnn\" (UID: \"67a2f33c-cef6-482f-b9bb-4c39342b7e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90d707-5df6477b5-rbbnn" Aug 04 16:45:42.213105 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:42.212947 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67a2f33c-cef6-482f-b9bb-4c39342b7e94-openshift-service-ca\") pod \"maas-api-e2e-shared-a-90d707-5df6477b5-rbbnn\" (UID: \"67a2f33c-cef6-482f-b9bb-4c39342b7e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90d707-5df6477b5-rbbnn" Aug 04 16:45:42.213105 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:42.213021 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc4n2\" (UniqueName: \"kubernetes.io/projected/67a2f33c-cef6-482f-b9bb-4c39342b7e94-kube-api-access-vc4n2\") pod \"maas-api-e2e-shared-a-90d707-5df6477b5-rbbnn\" (UID: \"67a2f33c-cef6-482f-b9bb-4c39342b7e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90d707-5df6477b5-rbbnn" Aug 04 16:45:42.314249 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:42.314213 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc4n2\" (UniqueName: \"kubernetes.io/projected/67a2f33c-cef6-482f-b9bb-4c39342b7e94-kube-api-access-vc4n2\") pod \"maas-api-e2e-shared-a-90d707-5df6477b5-rbbnn\" (UID: \"67a2f33c-cef6-482f-b9bb-4c39342b7e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90d707-5df6477b5-rbbnn" Aug 04 16:45:42.314451 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:42.314271 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67a2f33c-cef6-482f-b9bb-4c39342b7e94-maas-api-tls\") pod \"maas-api-e2e-shared-a-90d707-5df6477b5-rbbnn\" (UID: \"67a2f33c-cef6-482f-b9bb-4c39342b7e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90d707-5df6477b5-rbbnn" Aug 04 16:45:42.314451 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:42.314311 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67a2f33c-cef6-482f-b9bb-4c39342b7e94-openshift-service-ca\") pod \"maas-api-e2e-shared-a-90d707-5df6477b5-rbbnn\" (UID: \"67a2f33c-cef6-482f-b9bb-4c39342b7e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90d707-5df6477b5-rbbnn" Aug 04 16:45:42.314914 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:42.314896 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67a2f33c-cef6-482f-b9bb-4c39342b7e94-openshift-service-ca\") pod \"maas-api-e2e-shared-a-90d707-5df6477b5-rbbnn\" (UID: \"67a2f33c-cef6-482f-b9bb-4c39342b7e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90d707-5df6477b5-rbbnn" Aug 04 16:45:42.316678 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:42.316660 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67a2f33c-cef6-482f-b9bb-4c39342b7e94-maas-api-tls\") pod \"maas-api-e2e-shared-a-90d707-5df6477b5-rbbnn\" (UID: \"67a2f33c-cef6-482f-b9bb-4c39342b7e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90d707-5df6477b5-rbbnn" Aug 04 16:45:42.321728 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:42.321705 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc4n2\" (UniqueName: \"kubernetes.io/projected/67a2f33c-cef6-482f-b9bb-4c39342b7e94-kube-api-access-vc4n2\") pod \"maas-api-e2e-shared-a-90d707-5df6477b5-rbbnn\" (UID: \"67a2f33c-cef6-482f-b9bb-4c39342b7e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90d707-5df6477b5-rbbnn" Aug 04 16:45:42.435580 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:42.435474 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90d707-5df6477b5-rbbnn" Aug 04 16:45:42.569743 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:42.569701 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-90d707-5df6477b5-rbbnn"] Aug 04 16:45:42.572622 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:45:42.572594 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a2f33c_cef6_482f_b9bb_4c39342b7e94.slice/crio-a2ea11132846fb4b95b9e62abfaab7f466bfeb6471ac5059406c240c912cddf4 WatchSource:0}: Error finding container a2ea11132846fb4b95b9e62abfaab7f466bfeb6471ac5059406c240c912cddf4: Status 404 returned error can't find the container with id a2ea11132846fb4b95b9e62abfaab7f466bfeb6471ac5059406c240c912cddf4 Aug 04 16:45:43.146174 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:43.146150 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90d707-5df6477b5-rbbnn" event={"ID":"67a2f33c-cef6-482f-b9bb-4c39342b7e94","Type":"ContainerStarted","Data":"a2ea11132846fb4b95b9e62abfaab7f466bfeb6471ac5059406c240c912cddf4"} Aug 04 16:45:44.152628 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:44.152593 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90d707-5df6477b5-rbbnn" event={"ID":"67a2f33c-cef6-482f-b9bb-4c39342b7e94","Type":"ContainerStarted","Data":"65fd8a108c65bd4481034c4fcffd20aabe939ed260c3a33fff07e21669ad6c69"} Aug 04 16:45:44.153055 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:44.152726 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90d707-5df6477b5-rbbnn" Aug 04 16:45:44.168385 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:44.168334 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90d707-5df6477b5-rbbnn" podStartSLOduration=1.5980889390000002 podStartE2EDuration="2.168317679s" totalImagesPullingTime="570.22874ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:45:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:45:44.1669994 +0000 UTC m=+2943.584541526" watchObservedRunningTime="2026-08-04 16:45:44.168317679 +0000 UTC m=+2943.585859805" Aug 04 16:45:50.162119 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:45:50.162091 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90d707-5df6477b5-rbbnn" Aug 04 16:46:10.442886 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:46:10.442792 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w"] Aug 04 16:46:10.446273 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:46:10.446253 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w" Aug 04 16:46:10.450432 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:46:10.450410 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-5a3cb8-serving-cert\"" Aug 04 16:46:10.464722 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:46:10.464695 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w"] Aug 04 16:46:10.560859 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:46:10.560819 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm7j2\" (UniqueName: \"kubernetes.io/projected/6698886a-4153-4a7d-b552-a56daafe913f-kube-api-access-zm7j2\") pod \"maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w\" (UID: \"6698886a-4153-4a7d-b552-a56daafe913f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w" Aug 04 16:46:10.561053 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:46:10.560874 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6698886a-4153-4a7d-b552-a56daafe913f-maas-api-tls\") pod \"maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w\" (UID: \"6698886a-4153-4a7d-b552-a56daafe913f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w" Aug 04 16:46:10.561053 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:46:10.560941 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6698886a-4153-4a7d-b552-a56daafe913f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w\" (UID: \"6698886a-4153-4a7d-b552-a56daafe913f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w" Aug 04 16:46:10.661902 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:46:10.661861 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm7j2\" (UniqueName: \"kubernetes.io/projected/6698886a-4153-4a7d-b552-a56daafe913f-kube-api-access-zm7j2\") pod \"maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w\" (UID: \"6698886a-4153-4a7d-b552-a56daafe913f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w" Aug 04 16:46:10.662106 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:46:10.661924 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6698886a-4153-4a7d-b552-a56daafe913f-maas-api-tls\") pod \"maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w\" (UID: \"6698886a-4153-4a7d-b552-a56daafe913f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w" Aug 04 16:46:10.662106 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:46:10.661953 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6698886a-4153-4a7d-b552-a56daafe913f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w\" (UID: \"6698886a-4153-4a7d-b552-a56daafe913f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w" Aug 04 16:46:10.662106 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:46:10.662072 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-5a3cb8-serving-cert: secret "maas-api-e2e-shared-b-5a3cb8-serving-cert" not found Aug 04 16:46:10.662343 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:46:10.662154 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6698886a-4153-4a7d-b552-a56daafe913f-maas-api-tls podName:6698886a-4153-4a7d-b552-a56daafe913f nodeName:}" failed. No retries permitted until 2026-08-04 16:46:11.162134907 +0000 UTC m=+2970.579677011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6698886a-4153-4a7d-b552-a56daafe913f-maas-api-tls") pod "maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w" (UID: "6698886a-4153-4a7d-b552-a56daafe913f") : secret "maas-api-e2e-shared-b-5a3cb8-serving-cert" not found Aug 04 16:46:10.662659 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:46:10.662640 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6698886a-4153-4a7d-b552-a56daafe913f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w\" (UID: \"6698886a-4153-4a7d-b552-a56daafe913f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w" Aug 04 16:46:10.670131 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:46:10.670107 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm7j2\" (UniqueName: \"kubernetes.io/projected/6698886a-4153-4a7d-b552-a56daafe913f-kube-api-access-zm7j2\") pod \"maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w\" (UID: \"6698886a-4153-4a7d-b552-a56daafe913f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w" Aug 04 16:46:11.168635 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:46:11.168598 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6698886a-4153-4a7d-b552-a56daafe913f-maas-api-tls\") pod \"maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w\" (UID: \"6698886a-4153-4a7d-b552-a56daafe913f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w" Aug 04 16:46:11.171075 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:46:11.171056 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6698886a-4153-4a7d-b552-a56daafe913f-maas-api-tls\") pod \"maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w\" (UID: \"6698886a-4153-4a7d-b552-a56daafe913f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w" Aug 04 16:46:11.356804 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:46:11.356765 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w" Aug 04 16:46:11.487905 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:46:11.487875 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w"] Aug 04 16:46:11.490667 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:46:11.490639 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6698886a_4153_4a7d_b552_a56daafe913f.slice/crio-1a4f0585e0335c44de14548bb53ee0b9491a7a1520778bf4dd7272ca77aa32d7 WatchSource:0}: Error finding container 1a4f0585e0335c44de14548bb53ee0b9491a7a1520778bf4dd7272ca77aa32d7: Status 404 returned error can't find the container with id 1a4f0585e0335c44de14548bb53ee0b9491a7a1520778bf4dd7272ca77aa32d7 Aug 04 16:46:12.278736 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:46:12.278699 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w" event={"ID":"6698886a-4153-4a7d-b552-a56daafe913f","Type":"ContainerStarted","Data":"ea62f13f211f7f94b24a862ec9c7d1364cce6f59d638e80c53a96c4f10562b15"} Aug 04 16:46:12.278945 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:46:12.278755 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w" event={"ID":"6698886a-4153-4a7d-b552-a56daafe913f","Type":"ContainerStarted","Data":"1a4f0585e0335c44de14548bb53ee0b9491a7a1520778bf4dd7272ca77aa32d7"} Aug 04 16:46:12.278945 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:46:12.278808 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w" Aug 04 16:46:12.295617 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:46:12.295544 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w" podStartSLOduration=1.9882052080000001 podStartE2EDuration="2.295529528s" totalImagesPullingTime="307.32432ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:46:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:46:12.294104027 +0000 UTC m=+2971.711646165" watchObservedRunningTime="2026-08-04 16:46:12.295529528 +0000 UTC m=+2971.713071695" Aug 04 16:46:18.287791 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:46:18.287761 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w" Aug 04 16:46:41.175167 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:46:41.175134 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-gvb4l_c16f82ab-8340-49ee-b41e-5685d4592fd2/console-operator/2.log" Aug 04 16:46:41.179356 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:46:41.179332 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mtbc5_11f9cbee-a773-420b-9873-5c8cb7679dd8/ovn-acl-logging/0.log" Aug 04 16:46:41.181199 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:46:41.181180 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-gvb4l_c16f82ab-8340-49ee-b41e-5685d4592fd2/console-operator/2.log" Aug 04 16:46:41.184976 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:46:41.184959 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mtbc5_11f9cbee-a773-420b-9873-5c8cb7679dd8/ovn-acl-logging/0.log" Aug 04 16:50:57.607112 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:50:57.607074 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss"] Aug 04 16:50:57.610292 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:50:57.610275 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss" Aug 04 16:50:57.615123 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:50:57.615084 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-90d707\"/\"openshift-service-ca.crt\"" Aug 04 16:50:57.615123 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:50:57.615105 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-90d707\"/\"sub-test-model-90d707-kserve-self-signed-certs\"" Aug 04 16:50:57.615294 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:50:57.615088 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-90d707\"/\"kube-root-ca.crt\"" Aug 04 16:50:57.615294 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:50:57.615088 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-90d707\"/\"default-dockercfg-bgnk2\"" Aug 04 16:50:57.625925 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:50:57.625895 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss"] Aug 04 16:50:57.771216 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:50:57.771182 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07da3245-3936-417e-940b-6f5ee43839aa-model-cache\") pod \"sub-test-model-90d707-kserve-7ff7f97bc7-s54ss\" (UID: \"07da3245-3936-417e-940b-6f5ee43839aa\") " pod="ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss" Aug 04 16:50:57.771216 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:50:57.771228 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrzfm\" (UniqueName: \"kubernetes.io/projected/07da3245-3936-417e-940b-6f5ee43839aa-kube-api-access-qrzfm\") pod \"sub-test-model-90d707-kserve-7ff7f97bc7-s54ss\" (UID: \"07da3245-3936-417e-940b-6f5ee43839aa\") " pod="ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss" Aug 04 16:50:57.771466 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:50:57.771248 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07da3245-3936-417e-940b-6f5ee43839aa-home\") pod \"sub-test-model-90d707-kserve-7ff7f97bc7-s54ss\" (UID: \"07da3245-3936-417e-940b-6f5ee43839aa\") " pod="ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss" Aug 04 16:50:57.771466 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:50:57.771346 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07da3245-3936-417e-940b-6f5ee43839aa-tls-certs\") pod \"sub-test-model-90d707-kserve-7ff7f97bc7-s54ss\" (UID: \"07da3245-3936-417e-940b-6f5ee43839aa\") " pod="ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss" Aug 04 16:50:57.771466 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:50:57.771378 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07da3245-3936-417e-940b-6f5ee43839aa-tmp-dir\") pod \"sub-test-model-90d707-kserve-7ff7f97bc7-s54ss\" (UID: \"07da3245-3936-417e-940b-6f5ee43839aa\") " pod="ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss" Aug 04 16:50:57.771466 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:50:57.771407 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07da3245-3936-417e-940b-6f5ee43839aa-dshm\") pod \"sub-test-model-90d707-kserve-7ff7f97bc7-s54ss\" (UID: \"07da3245-3936-417e-940b-6f5ee43839aa\") " pod="ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss" Aug 04 16:50:57.872035 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:50:57.871945 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07da3245-3936-417e-940b-6f5ee43839aa-tmp-dir\") pod \"sub-test-model-90d707-kserve-7ff7f97bc7-s54ss\" (UID: \"07da3245-3936-417e-940b-6f5ee43839aa\") " pod="ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss" Aug 04 16:50:57.872035 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:50:57.872002 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07da3245-3936-417e-940b-6f5ee43839aa-dshm\") pod \"sub-test-model-90d707-kserve-7ff7f97bc7-s54ss\" (UID: \"07da3245-3936-417e-940b-6f5ee43839aa\") " pod="ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss" Aug 04 16:50:57.872232 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:50:57.872059 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07da3245-3936-417e-940b-6f5ee43839aa-model-cache\") pod \"sub-test-model-90d707-kserve-7ff7f97bc7-s54ss\" (UID: \"07da3245-3936-417e-940b-6f5ee43839aa\") " pod="ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss" Aug 04 16:50:57.872232 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:50:57.872099 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrzfm\" (UniqueName: \"kubernetes.io/projected/07da3245-3936-417e-940b-6f5ee43839aa-kube-api-access-qrzfm\") pod \"sub-test-model-90d707-kserve-7ff7f97bc7-s54ss\" (UID: \"07da3245-3936-417e-940b-6f5ee43839aa\") " pod="ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss" Aug 04 16:50:57.872232 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:50:57.872131 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07da3245-3936-417e-940b-6f5ee43839aa-home\") pod \"sub-test-model-90d707-kserve-7ff7f97bc7-s54ss\" (UID: \"07da3245-3936-417e-940b-6f5ee43839aa\") " pod="ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss" Aug 04 16:50:57.872232 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:50:57.872183 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07da3245-3936-417e-940b-6f5ee43839aa-tls-certs\") pod \"sub-test-model-90d707-kserve-7ff7f97bc7-s54ss\" (UID: \"07da3245-3936-417e-940b-6f5ee43839aa\") " pod="ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss" Aug 04 16:50:57.872435 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:50:57.872378 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07da3245-3936-417e-940b-6f5ee43839aa-tmp-dir\") pod \"sub-test-model-90d707-kserve-7ff7f97bc7-s54ss\" (UID: \"07da3245-3936-417e-940b-6f5ee43839aa\") " pod="ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss" Aug 04 16:50:57.872489 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:50:57.872428 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07da3245-3936-417e-940b-6f5ee43839aa-model-cache\") pod \"sub-test-model-90d707-kserve-7ff7f97bc7-s54ss\" (UID: \"07da3245-3936-417e-940b-6f5ee43839aa\") " pod="ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss" Aug 04 16:50:57.872544 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:50:57.872500 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07da3245-3936-417e-940b-6f5ee43839aa-home\") pod \"sub-test-model-90d707-kserve-7ff7f97bc7-s54ss\" (UID: \"07da3245-3936-417e-940b-6f5ee43839aa\") " pod="ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss" Aug 04 16:50:57.874301 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:50:57.874265 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07da3245-3936-417e-940b-6f5ee43839aa-dshm\") pod \"sub-test-model-90d707-kserve-7ff7f97bc7-s54ss\" (UID: \"07da3245-3936-417e-940b-6f5ee43839aa\") " pod="ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss" Aug 04 16:50:57.874691 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:50:57.874671 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07da3245-3936-417e-940b-6f5ee43839aa-tls-certs\") pod \"sub-test-model-90d707-kserve-7ff7f97bc7-s54ss\" (UID: \"07da3245-3936-417e-940b-6f5ee43839aa\") " pod="ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss" Aug 04 16:50:57.883698 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:50:57.883677 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrzfm\" (UniqueName: \"kubernetes.io/projected/07da3245-3936-417e-940b-6f5ee43839aa-kube-api-access-qrzfm\") pod \"sub-test-model-90d707-kserve-7ff7f97bc7-s54ss\" (UID: \"07da3245-3936-417e-940b-6f5ee43839aa\") " pod="ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss" Aug 04 16:50:57.920391 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:50:57.920354 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss" Aug 04 16:50:58.061610 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:50:58.061574 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss"] Aug 04 16:50:58.065012 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:50:58.064988 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07da3245_3936_417e_940b_6f5ee43839aa.slice/crio-f34fc131f65745eabf5439b45815debc9b5e1ecadbf92811d5d9c8aff9ad0181 WatchSource:0}: Error finding container f34fc131f65745eabf5439b45815debc9b5e1ecadbf92811d5d9c8aff9ad0181: Status 404 returned error can't find the container with id f34fc131f65745eabf5439b45815debc9b5e1ecadbf92811d5d9c8aff9ad0181 Aug 04 16:50:58.501627 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:50:58.501588 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss" event={"ID":"07da3245-3936-417e-940b-6f5ee43839aa","Type":"ContainerStarted","Data":"6c663d49a801ff188e550b75262f3f70000b72c342de760e4a34eb54008ec6bc"} Aug 04 16:50:58.501627 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:50:58.501632 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss" event={"ID":"07da3245-3936-417e-940b-6f5ee43839aa","Type":"ContainerStarted","Data":"f34fc131f65745eabf5439b45815debc9b5e1ecadbf92811d5d9c8aff9ad0181"} Aug 04 16:50:58.528209 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:50:58.528162 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss" podStartSLOduration=1.528147609 podStartE2EDuration="1.528147609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:50:58.527221503 +0000 UTC m=+3257.944763634" watchObservedRunningTime="2026-08-04 16:50:58.528147609 +0000 UTC m=+3257.945689776" Aug 04 16:51:07.920926 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:51:07.920888 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss" Aug 04 16:51:07.921387 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:51:07.920959 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss" Aug 04 16:51:07.933969 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:51:07.933938 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss" Aug 04 16:51:08.560642 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:51:08.560605 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss" Aug 04 16:51:41.199337 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:51:41.199307 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-gvb4l_c16f82ab-8340-49ee-b41e-5685d4592fd2/console-operator/2.log" Aug 04 16:51:41.203667 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:51:41.203640 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mtbc5_11f9cbee-a773-420b-9873-5c8cb7679dd8/ovn-acl-logging/0.log" Aug 04 16:51:41.205861 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:51:41.205831 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-gvb4l_c16f82ab-8340-49ee-b41e-5685d4592fd2/console-operator/2.log" Aug 04 16:51:41.209794 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:51:41.209773 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mtbc5_11f9cbee-a773-420b-9873-5c8cb7679dd8/ovn-acl-logging/0.log" Aug 04 16:52:49.789327 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:52:49.789286 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss"] Aug 04 16:53:04.808074 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:53:04.808033 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss" podUID="07da3245-3936-417e-940b-6f5ee43839aa" containerName="main" containerID="cri-o://6c663d49a801ff188e550b75262f3f70000b72c342de760e4a34eb54008ec6bc" gracePeriod=45 Aug 04 16:53:05.046758 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:53:05.046730 2573 generic.go:356] "Generic (PLEG): container finished" podID="07da3245-3936-417e-940b-6f5ee43839aa" containerID="6c663d49a801ff188e550b75262f3f70000b72c342de760e4a34eb54008ec6bc" exitCode=0 Aug 04 16:53:05.046898 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:53:05.046775 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss" event={"ID":"07da3245-3936-417e-940b-6f5ee43839aa","Type":"ContainerDied","Data":"6c663d49a801ff188e550b75262f3f70000b72c342de760e4a34eb54008ec6bc"} Aug 04 16:53:05.046898 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:53:05.046803 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss" event={"ID":"07da3245-3936-417e-940b-6f5ee43839aa","Type":"ContainerDied","Data":"f34fc131f65745eabf5439b45815debc9b5e1ecadbf92811d5d9c8aff9ad0181"} Aug 04 16:53:05.046898 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:53:05.046814 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f34fc131f65745eabf5439b45815debc9b5e1ecadbf92811d5d9c8aff9ad0181" Aug 04 16:53:05.048370 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:53:05.048352 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss" Aug 04 16:53:05.072859 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:53:05.072782 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07da3245-3936-417e-940b-6f5ee43839aa-home\") pod \"07da3245-3936-417e-940b-6f5ee43839aa\" (UID: \"07da3245-3936-417e-940b-6f5ee43839aa\") " Aug 04 16:53:05.072859 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:53:05.072818 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07da3245-3936-417e-940b-6f5ee43839aa-tls-certs\") pod \"07da3245-3936-417e-940b-6f5ee43839aa\" (UID: \"07da3245-3936-417e-940b-6f5ee43839aa\") " Aug 04 16:53:05.078608 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:53:05.073169 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07da3245-3936-417e-940b-6f5ee43839aa-tmp-dir\") pod \"07da3245-3936-417e-940b-6f5ee43839aa\" (UID: \"07da3245-3936-417e-940b-6f5ee43839aa\") " Aug 04 16:53:05.078608 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:53:05.073340 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07da3245-3936-417e-940b-6f5ee43839aa-model-cache\") pod \"07da3245-3936-417e-940b-6f5ee43839aa\" (UID: \"07da3245-3936-417e-940b-6f5ee43839aa\") " Aug 04 16:53:05.078608 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:53:05.073349 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07da3245-3936-417e-940b-6f5ee43839aa-home" (OuterVolumeSpecName: "home") pod "07da3245-3936-417e-940b-6f5ee43839aa" (UID: "07da3245-3936-417e-940b-6f5ee43839aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:53:05.078608 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:53:05.073381 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07da3245-3936-417e-940b-6f5ee43839aa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "07da3245-3936-417e-940b-6f5ee43839aa" (UID: "07da3245-3936-417e-940b-6f5ee43839aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:53:05.078608 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:53:05.073398 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07da3245-3936-417e-940b-6f5ee43839aa-dshm\") pod \"07da3245-3936-417e-940b-6f5ee43839aa\" (UID: \"07da3245-3936-417e-940b-6f5ee43839aa\") " Aug 04 16:53:05.078608 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:53:05.073446 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrzfm\" (UniqueName: \"kubernetes.io/projected/07da3245-3936-417e-940b-6f5ee43839aa-kube-api-access-qrzfm\") pod \"07da3245-3936-417e-940b-6f5ee43839aa\" (UID: \"07da3245-3936-417e-940b-6f5ee43839aa\") " Aug 04 16:53:05.078608 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:53:05.073603 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07da3245-3936-417e-940b-6f5ee43839aa-model-cache" (OuterVolumeSpecName: "model-cache") pod "07da3245-3936-417e-940b-6f5ee43839aa" (UID: "07da3245-3936-417e-940b-6f5ee43839aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:53:05.078608 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:53:05.075260 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07da3245-3936-417e-940b-6f5ee43839aa-model-cache\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:53:05.078608 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:53:05.075284 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07da3245-3936-417e-940b-6f5ee43839aa-home\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:53:05.078608 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:53:05.075465 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07da3245-3936-417e-940b-6f5ee43839aa-tmp-dir\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:53:05.078608 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:53:05.077261 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07da3245-3936-417e-940b-6f5ee43839aa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "07da3245-3936-417e-940b-6f5ee43839aa" (UID: "07da3245-3936-417e-940b-6f5ee43839aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:53:05.082076 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:53:05.081213 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07da3245-3936-417e-940b-6f5ee43839aa-dshm" (OuterVolumeSpecName: "dshm") pod "07da3245-3936-417e-940b-6f5ee43839aa" (UID: "07da3245-3936-417e-940b-6f5ee43839aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:53:05.082530 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:53:05.082502 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07da3245-3936-417e-940b-6f5ee43839aa-kube-api-access-qrzfm" (OuterVolumeSpecName: "kube-api-access-qrzfm") pod "07da3245-3936-417e-940b-6f5ee43839aa" (UID: "07da3245-3936-417e-940b-6f5ee43839aa"). InnerVolumeSpecName "kube-api-access-qrzfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:53:05.176460 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:53:05.176427 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrzfm\" (UniqueName: \"kubernetes.io/projected/07da3245-3936-417e-940b-6f5ee43839aa-kube-api-access-qrzfm\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:53:05.176460 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:53:05.176455 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07da3245-3936-417e-940b-6f5ee43839aa-tls-certs\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:53:05.176460 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:53:05.176468 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07da3245-3936-417e-940b-6f5ee43839aa-dshm\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:53:06.051365 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:53:06.051331 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss" Aug 04 16:53:06.065291 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:53:06.065263 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss"] Aug 04 16:53:06.072041 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:53:06.072008 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-90d707/sub-test-model-90d707-kserve-7ff7f97bc7-s54ss"] Aug 04 16:53:07.078108 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:53:07.078076 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07da3245-3936-417e-940b-6f5ee43839aa" path="/var/lib/kubelet/pods/07da3245-3936-417e-940b-6f5ee43839aa/volumes" Aug 04 16:54:50.283760 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:50.283719 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7355-140050664032816"] Aug 04 16:54:50.284220 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:50.284029 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07da3245-3936-417e-940b-6f5ee43839aa" containerName="main" Aug 04 16:54:50.284220 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:50.284041 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="07da3245-3936-417e-940b-6f5ee43839aa" containerName="main" Aug 04 16:54:50.284220 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:50.284102 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="07da3245-3936-417e-940b-6f5ee43839aa" containerName="main" Aug 04 16:54:50.287004 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:50.286988 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7355-140050664032816" Aug 04 16:54:50.289068 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:50.289046 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-gstnk\"" Aug 04 16:54:50.295825 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:50.294345 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7355-140050664032816"] Aug 04 16:54:50.378312 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:50.378273 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd48r\" (UniqueName: \"kubernetes.io/projected/0e290b52-a86b-4ebe-a831-a910a07e55c3-kube-api-access-kd48r\") pod \"test-curl-7355-140050664032816\" (UID: \"0e290b52-a86b-4ebe-a831-a910a07e55c3\") " pod="opendatahub/test-curl-7355-140050664032816" Aug 04 16:54:50.479691 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:50.479649 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd48r\" (UniqueName: \"kubernetes.io/projected/0e290b52-a86b-4ebe-a831-a910a07e55c3-kube-api-access-kd48r\") pod \"test-curl-7355-140050664032816\" (UID: \"0e290b52-a86b-4ebe-a831-a910a07e55c3\") " pod="opendatahub/test-curl-7355-140050664032816" Aug 04 16:54:50.487701 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:50.487674 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd48r\" (UniqueName: \"kubernetes.io/projected/0e290b52-a86b-4ebe-a831-a910a07e55c3-kube-api-access-kd48r\") pod \"test-curl-7355-140050664032816\" (UID: \"0e290b52-a86b-4ebe-a831-a910a07e55c3\") " pod="opendatahub/test-curl-7355-140050664032816" Aug 04 16:54:50.605523 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:50.605445 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7355-140050664032816" Aug 04 16:54:50.742045 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:50.742011 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7355-140050664032816"] Aug 04 16:54:50.745108 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:54:50.745079 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e290b52_a86b_4ebe_a831_a910a07e55c3.slice/crio-dc3c04cba4aa0b113c2e0ba2f3862e40b17cd7098f5f171258435904ff75f622 WatchSource:0}: Error finding container dc3c04cba4aa0b113c2e0ba2f3862e40b17cd7098f5f171258435904ff75f622: Status 404 returned error can't find the container with id dc3c04cba4aa0b113c2e0ba2f3862e40b17cd7098f5f171258435904ff75f622 Aug 04 16:54:50.746606 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:50.746584 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 16:54:51.502059 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:51.502025 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7355-140050664032816" event={"ID":"0e290b52-a86b-4ebe-a831-a910a07e55c3","Type":"ContainerStarted","Data":"c84ee21e7669ee6404a825cd05bb13bb8c828300ad7229f233b75a65371d18d8"} Aug 04 16:54:51.502059 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:51.502072 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7355-140050664032816" event={"ID":"0e290b52-a86b-4ebe-a831-a910a07e55c3","Type":"ContainerStarted","Data":"dc3c04cba4aa0b113c2e0ba2f3862e40b17cd7098f5f171258435904ff75f622"} Aug 04 16:54:51.516795 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:51.516744 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7355-140050664032816" podStartSLOduration=0.840396233 podStartE2EDuration="1.516727952s" totalImagesPullingTime="676.331719ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:54:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:54:51.515506297 +0000 UTC m=+3490.933048424" watchObservedRunningTime="2026-08-04 16:54:51.516727952 +0000 UTC m=+3490.934270078" Aug 04 16:54:51.626955 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:51.626926 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7355-140050664032816_0e290b52-a86b-4ebe-a831-a910a07e55c3/test-curl-7355-140050664032816/0.log" Aug 04 16:54:52.507463 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:52.507428 2573 generic.go:356] "Generic (PLEG): container finished" podID="0e290b52-a86b-4ebe-a831-a910a07e55c3" containerID="c84ee21e7669ee6404a825cd05bb13bb8c828300ad7229f233b75a65371d18d8" exitCode=0 Aug 04 16:54:52.507872 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:52.507495 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7355-140050664032816" event={"ID":"0e290b52-a86b-4ebe-a831-a910a07e55c3","Type":"ContainerDied","Data":"c84ee21e7669ee6404a825cd05bb13bb8c828300ad7229f233b75a65371d18d8"} Aug 04 16:54:53.639082 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:53.639055 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7355-140050664032816" Aug 04 16:54:53.681951 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:53.681916 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7355-140050664032816"] Aug 04 16:54:53.684756 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:53.684731 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7355-140050664032816"] Aug 04 16:54:53.707116 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:53.707093 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kd48r\" (UniqueName: \"kubernetes.io/projected/0e290b52-a86b-4ebe-a831-a910a07e55c3-kube-api-access-kd48r\") pod \"0e290b52-a86b-4ebe-a831-a910a07e55c3\" (UID: \"0e290b52-a86b-4ebe-a831-a910a07e55c3\") " Aug 04 16:54:53.709161 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:53.709127 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e290b52-a86b-4ebe-a831-a910a07e55c3-kube-api-access-kd48r" (OuterVolumeSpecName: "kube-api-access-kd48r") pod "0e290b52-a86b-4ebe-a831-a910a07e55c3" (UID: "0e290b52-a86b-4ebe-a831-a910a07e55c3"). InnerVolumeSpecName "kube-api-access-kd48r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:54:53.807836 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:53.807747 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd48r\" (UniqueName: \"kubernetes.io/projected/0e290b52-a86b-4ebe-a831-a910a07e55c3-kube-api-access-kd48r\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:54:53.808584 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:53.808525 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7355-140050664030512"] Aug 04 16:54:53.808922 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:53.808905 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e290b52-a86b-4ebe-a831-a910a07e55c3" containerName="test-curl-7355-140050664032816" Aug 04 16:54:53.809003 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:53.808925 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e290b52-a86b-4ebe-a831-a910a07e55c3" containerName="test-curl-7355-140050664032816" Aug 04 16:54:53.809059 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:53.809034 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e290b52-a86b-4ebe-a831-a910a07e55c3" containerName="test-curl-7355-140050664032816" Aug 04 16:54:53.811856 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:53.811836 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7355-140050664030512" Aug 04 16:54:53.815624 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:53.815604 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7355-140050664030512"] Aug 04 16:54:53.908403 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:53.908368 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpqpm\" (UniqueName: \"kubernetes.io/projected/645f1547-1a57-48f3-8cda-625ce436560f-kube-api-access-xpqpm\") pod \"test-curl-7355-140050664030512\" (UID: \"645f1547-1a57-48f3-8cda-625ce436560f\") " pod="opendatahub/test-curl-7355-140050664030512" Aug 04 16:54:54.008877 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:54.008838 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpqpm\" (UniqueName: \"kubernetes.io/projected/645f1547-1a57-48f3-8cda-625ce436560f-kube-api-access-xpqpm\") pod \"test-curl-7355-140050664030512\" (UID: \"645f1547-1a57-48f3-8cda-625ce436560f\") " pod="opendatahub/test-curl-7355-140050664030512" Aug 04 16:54:54.017319 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:54.017291 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpqpm\" (UniqueName: \"kubernetes.io/projected/645f1547-1a57-48f3-8cda-625ce436560f-kube-api-access-xpqpm\") pod \"test-curl-7355-140050664030512\" (UID: \"645f1547-1a57-48f3-8cda-625ce436560f\") " pod="opendatahub/test-curl-7355-140050664030512" Aug 04 16:54:54.122417 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:54.122329 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7355-140050664030512" Aug 04 16:54:54.251219 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:54.251193 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7355-140050664030512"] Aug 04 16:54:54.253750 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:54:54.253723 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod645f1547_1a57_48f3_8cda_625ce436560f.slice/crio-8f487998406484b5f9c93e6652731c1e0b31a24edfda951b90a841941d2b5e80 WatchSource:0}: Error finding container 8f487998406484b5f9c93e6652731c1e0b31a24edfda951b90a841941d2b5e80: Status 404 returned error can't find the container with id 8f487998406484b5f9c93e6652731c1e0b31a24edfda951b90a841941d2b5e80 Aug 04 16:54:54.516960 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:54.516935 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7355-140050664032816" Aug 04 16:54:54.516960 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:54.516969 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc3c04cba4aa0b113c2e0ba2f3862e40b17cd7098f5f171258435904ff75f622" Aug 04 16:54:54.518007 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:54.517983 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7355-140050664030512" event={"ID":"645f1547-1a57-48f3-8cda-625ce436560f","Type":"ContainerStarted","Data":"8f487998406484b5f9c93e6652731c1e0b31a24edfda951b90a841941d2b5e80"} Aug 04 16:54:54.527544 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:54:54.527511 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7355-140050664032816\" is forbidden: User \"system:node:ip-10-0-141-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-210.ec2.internal' and this object" podUID="0e290b52-a86b-4ebe-a831-a910a07e55c3" pod="opendatahub/test-curl-7355-140050664032816" Aug 04 16:54:55.078138 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:55.078102 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e290b52-a86b-4ebe-a831-a910a07e55c3" path="/var/lib/kubelet/pods/0e290b52-a86b-4ebe-a831-a910a07e55c3/volumes" Aug 04 16:54:55.525378 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:55.525344 2573 generic.go:356] "Generic (PLEG): container finished" podID="645f1547-1a57-48f3-8cda-625ce436560f" containerID="264ab8bf4d09f20fe4296b4989916759caa9948abef96a60edef8f2782b507a2" exitCode=0 Aug 04 16:54:55.525604 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:55.525434 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7355-140050664030512" event={"ID":"645f1547-1a57-48f3-8cda-625ce436560f","Type":"ContainerDied","Data":"264ab8bf4d09f20fe4296b4989916759caa9948abef96a60edef8f2782b507a2"} Aug 04 16:54:56.662876 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:56.662851 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7355-140050664030512" Aug 04 16:54:56.681909 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:56.681865 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7355-140050664030512_645f1547-1a57-48f3-8cda-625ce436560f/test-curl-7355-140050664030512/0.log" Aug 04 16:54:56.705299 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:56.705257 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7355-140050664030512"] Aug 04 16:54:56.707329 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:56.707300 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7355-140050664030512"] Aug 04 16:54:56.738844 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:56.738816 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpqpm\" (UniqueName: \"kubernetes.io/projected/645f1547-1a57-48f3-8cda-625ce436560f-kube-api-access-xpqpm\") pod \"645f1547-1a57-48f3-8cda-625ce436560f\" (UID: \"645f1547-1a57-48f3-8cda-625ce436560f\") " Aug 04 16:54:56.740974 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:56.740951 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/645f1547-1a57-48f3-8cda-625ce436560f-kube-api-access-xpqpm" (OuterVolumeSpecName: "kube-api-access-xpqpm") pod "645f1547-1a57-48f3-8cda-625ce436560f" (UID: "645f1547-1a57-48f3-8cda-625ce436560f"). InnerVolumeSpecName "kube-api-access-xpqpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:54:56.840120 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:56.840032 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpqpm\" (UniqueName: \"kubernetes.io/projected/645f1547-1a57-48f3-8cda-625ce436560f-kube-api-access-xpqpm\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:54:56.877668 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:56.877633 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7355-140050664000800"] Aug 04 16:54:56.877981 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:56.877968 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="645f1547-1a57-48f3-8cda-625ce436560f" containerName="test-curl-7355-140050664030512" Aug 04 16:54:56.878029 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:56.877983 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="645f1547-1a57-48f3-8cda-625ce436560f" containerName="test-curl-7355-140050664030512" Aug 04 16:54:56.878066 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:56.878051 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="645f1547-1a57-48f3-8cda-625ce436560f" containerName="test-curl-7355-140050664030512" Aug 04 16:54:56.880883 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:56.880864 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7355-140050664000800" Aug 04 16:54:56.885371 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:56.885335 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7355-140050664000800"] Aug 04 16:54:56.940625 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:56.940588 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75qbw\" (UniqueName: \"kubernetes.io/projected/6dce707b-3cae-4a43-8ed6-108c81c63538-kube-api-access-75qbw\") pod \"test-curl-7355-140050664000800\" (UID: \"6dce707b-3cae-4a43-8ed6-108c81c63538\") " pod="opendatahub/test-curl-7355-140050664000800" Aug 04 16:54:57.041706 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:57.041675 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75qbw\" (UniqueName: \"kubernetes.io/projected/6dce707b-3cae-4a43-8ed6-108c81c63538-kube-api-access-75qbw\") pod \"test-curl-7355-140050664000800\" (UID: \"6dce707b-3cae-4a43-8ed6-108c81c63538\") " pod="opendatahub/test-curl-7355-140050664000800" Aug 04 16:54:57.049899 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:57.049873 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75qbw\" (UniqueName: \"kubernetes.io/projected/6dce707b-3cae-4a43-8ed6-108c81c63538-kube-api-access-75qbw\") pod \"test-curl-7355-140050664000800\" (UID: \"6dce707b-3cae-4a43-8ed6-108c81c63538\") " pod="opendatahub/test-curl-7355-140050664000800" Aug 04 16:54:57.078314 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:57.078281 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="645f1547-1a57-48f3-8cda-625ce436560f" path="/var/lib/kubelet/pods/645f1547-1a57-48f3-8cda-625ce436560f/volumes" Aug 04 16:54:57.191454 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:57.191367 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7355-140050664000800" Aug 04 16:54:57.533761 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:57.533461 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7355-140050664000800"] Aug 04 16:54:57.535165 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:57.535144 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7355-140050664030512" Aug 04 16:54:57.535165 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:57.535156 2573 scope.go:117] "RemoveContainer" containerID="264ab8bf4d09f20fe4296b4989916759caa9948abef96a60edef8f2782b507a2" Aug 04 16:54:57.536258 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:54:57.536231 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6dce707b_3cae_4a43_8ed6_108c81c63538.slice/crio-bf46f6cac781acffa0b15e092ee27443f8fb25130221620d9157876cdaead18f WatchSource:0}: Error finding container bf46f6cac781acffa0b15e092ee27443f8fb25130221620d9157876cdaead18f: Status 404 returned error can't find the container with id bf46f6cac781acffa0b15e092ee27443f8fb25130221620d9157876cdaead18f Aug 04 16:54:58.542686 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:58.542646 2573 generic.go:356] "Generic (PLEG): container finished" podID="6dce707b-3cae-4a43-8ed6-108c81c63538" containerID="d97caa8da132edf1a9ce8c499bbc0325c4ff85f87b8709e6a0e17c4148f5a238" exitCode=0 Aug 04 16:54:58.543087 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:58.542740 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7355-140050664000800" event={"ID":"6dce707b-3cae-4a43-8ed6-108c81c63538","Type":"ContainerDied","Data":"d97caa8da132edf1a9ce8c499bbc0325c4ff85f87b8709e6a0e17c4148f5a238"} Aug 04 16:54:58.543087 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:58.542786 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7355-140050664000800" event={"ID":"6dce707b-3cae-4a43-8ed6-108c81c63538","Type":"ContainerStarted","Data":"bf46f6cac781acffa0b15e092ee27443f8fb25130221620d9157876cdaead18f"} Aug 04 16:54:59.679263 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:59.679238 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7355-140050664000800" Aug 04 16:54:59.699002 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:59.698966 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7355-140050664000800_6dce707b-3cae-4a43-8ed6-108c81c63538/test-curl-7355-140050664000800/0.log" Aug 04 16:54:59.723372 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:59.723335 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7355-140050664000800"] Aug 04 16:54:59.725216 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:59.725185 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7355-140050664000800"] Aug 04 16:54:59.767457 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:59.767429 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75qbw\" (UniqueName: \"kubernetes.io/projected/6dce707b-3cae-4a43-8ed6-108c81c63538-kube-api-access-75qbw\") pod \"6dce707b-3cae-4a43-8ed6-108c81c63538\" (UID: \"6dce707b-3cae-4a43-8ed6-108c81c63538\") " Aug 04 16:54:59.769515 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:59.769482 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dce707b-3cae-4a43-8ed6-108c81c63538-kube-api-access-75qbw" (OuterVolumeSpecName: "kube-api-access-75qbw") pod "6dce707b-3cae-4a43-8ed6-108c81c63538" (UID: "6dce707b-3cae-4a43-8ed6-108c81c63538"). InnerVolumeSpecName "kube-api-access-75qbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:54:59.847614 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:59.847530 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7355-140050664000800"] Aug 04 16:54:59.847885 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:59.847871 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dce707b-3cae-4a43-8ed6-108c81c63538" containerName="test-curl-7355-140050664000800" Aug 04 16:54:59.847925 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:59.847887 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dce707b-3cae-4a43-8ed6-108c81c63538" containerName="test-curl-7355-140050664000800" Aug 04 16:54:59.847969 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:59.847959 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dce707b-3cae-4a43-8ed6-108c81c63538" containerName="test-curl-7355-140050664000800" Aug 04 16:54:59.850793 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:59.850776 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7355-140050664000800" Aug 04 16:54:59.857393 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:59.857363 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7355-140050664000800"] Aug 04 16:54:59.868959 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:59.868929 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75qbw\" (UniqueName: \"kubernetes.io/projected/6dce707b-3cae-4a43-8ed6-108c81c63538-kube-api-access-75qbw\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:54:59.969805 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:54:59.969761 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t46d2\" (UniqueName: \"kubernetes.io/projected/f45a0691-3a93-4466-b804-831455027174-kube-api-access-t46d2\") pod \"test-curl-7355-140050664000800\" (UID: \"f45a0691-3a93-4466-b804-831455027174\") " pod="opendatahub/test-curl-7355-140050664000800" Aug 04 16:55:00.070367 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:00.070331 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t46d2\" (UniqueName: \"kubernetes.io/projected/f45a0691-3a93-4466-b804-831455027174-kube-api-access-t46d2\") pod \"test-curl-7355-140050664000800\" (UID: \"f45a0691-3a93-4466-b804-831455027174\") " pod="opendatahub/test-curl-7355-140050664000800" Aug 04 16:55:00.078748 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:00.078712 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t46d2\" (UniqueName: \"kubernetes.io/projected/f45a0691-3a93-4466-b804-831455027174-kube-api-access-t46d2\") pod \"test-curl-7355-140050664000800\" (UID: \"f45a0691-3a93-4466-b804-831455027174\") " pod="opendatahub/test-curl-7355-140050664000800" Aug 04 16:55:00.161291 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:00.161219 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7355-140050664000800" Aug 04 16:55:00.293983 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:00.293948 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7355-140050664000800"] Aug 04 16:55:00.297934 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:55:00.297911 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf45a0691_3a93_4466_b804_831455027174.slice/crio-b9b50eaf1e7ff81a6bc1194449df549578e8c26d4c920879e28af0170aad24a5 WatchSource:0}: Error finding container b9b50eaf1e7ff81a6bc1194449df549578e8c26d4c920879e28af0170aad24a5: Status 404 returned error can't find the container with id b9b50eaf1e7ff81a6bc1194449df549578e8c26d4c920879e28af0170aad24a5 Aug 04 16:55:00.551713 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:00.551684 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf46f6cac781acffa0b15e092ee27443f8fb25130221620d9157876cdaead18f" Aug 04 16:55:00.551713 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:00.551688 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7355-140050664000800" Aug 04 16:55:00.552827 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:00.552800 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7355-140050664000800" event={"ID":"f45a0691-3a93-4466-b804-831455027174","Type":"ContainerStarted","Data":"b9b50eaf1e7ff81a6bc1194449df549578e8c26d4c920879e28af0170aad24a5"} Aug 04 16:55:00.563299 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:00.563267 2573 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7355-140050664000800" oldPodUID="6dce707b-3cae-4a43-8ed6-108c81c63538" podUID="f45a0691-3a93-4466-b804-831455027174" Aug 04 16:55:01.078479 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:01.078444 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dce707b-3cae-4a43-8ed6-108c81c63538" path="/var/lib/kubelet/pods/6dce707b-3cae-4a43-8ed6-108c81c63538/volumes" Aug 04 16:55:01.558674 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:01.558638 2573 generic.go:356] "Generic (PLEG): container finished" podID="f45a0691-3a93-4466-b804-831455027174" containerID="59f1d1988feac044fd001e28d0c6216caa4caffdec1ee89a5644c8220f432020" exitCode=0 Aug 04 16:55:01.558859 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:01.558727 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7355-140050664000800" event={"ID":"f45a0691-3a93-4466-b804-831455027174","Type":"ContainerDied","Data":"59f1d1988feac044fd001e28d0c6216caa4caffdec1ee89a5644c8220f432020"} Aug 04 16:55:02.696280 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:02.696258 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7355-140050664000800" Aug 04 16:55:02.716623 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:02.716581 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7355-140050664000800_f45a0691-3a93-4466-b804-831455027174/test-curl-7355-140050664000800/0.log" Aug 04 16:55:02.742676 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:02.742645 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7355-140050664000800"] Aug 04 16:55:02.746758 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:02.746734 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7355-140050664000800"] Aug 04 16:55:02.793902 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:02.793873 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t46d2\" (UniqueName: \"kubernetes.io/projected/f45a0691-3a93-4466-b804-831455027174-kube-api-access-t46d2\") pod \"f45a0691-3a93-4466-b804-831455027174\" (UID: \"f45a0691-3a93-4466-b804-831455027174\") " Aug 04 16:55:02.795969 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:02.795946 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f45a0691-3a93-4466-b804-831455027174-kube-api-access-t46d2" (OuterVolumeSpecName: "kube-api-access-t46d2") pod "f45a0691-3a93-4466-b804-831455027174" (UID: "f45a0691-3a93-4466-b804-831455027174"). InnerVolumeSpecName "kube-api-access-t46d2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:55:02.895266 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:02.895182 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t46d2\" (UniqueName: \"kubernetes.io/projected/f45a0691-3a93-4466-b804-831455027174-kube-api-access-t46d2\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:55:02.900731 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:02.900700 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7355-140050664075600"] Aug 04 16:55:02.901115 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:02.901096 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f45a0691-3a93-4466-b804-831455027174" containerName="test-curl-7355-140050664000800" Aug 04 16:55:02.901198 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:02.901118 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f45a0691-3a93-4466-b804-831455027174" containerName="test-curl-7355-140050664000800" Aug 04 16:55:02.901198 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:02.901187 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f45a0691-3a93-4466-b804-831455027174" containerName="test-curl-7355-140050664000800" Aug 04 16:55:02.904061 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:02.904043 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7355-140050664075600" Aug 04 16:55:02.911536 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:02.911512 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7355-140050664075600"] Aug 04 16:55:02.996105 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:02.996065 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wplbb\" (UniqueName: \"kubernetes.io/projected/d679e459-1f2f-4384-b4ac-cc45d0a6b62d-kube-api-access-wplbb\") pod \"test-curl-7355-140050664075600\" (UID: \"d679e459-1f2f-4384-b4ac-cc45d0a6b62d\") " pod="opendatahub/test-curl-7355-140050664075600" Aug 04 16:55:03.079320 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:03.079288 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f45a0691-3a93-4466-b804-831455027174" path="/var/lib/kubelet/pods/f45a0691-3a93-4466-b804-831455027174/volumes" Aug 04 16:55:03.096606 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:03.096582 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wplbb\" (UniqueName: \"kubernetes.io/projected/d679e459-1f2f-4384-b4ac-cc45d0a6b62d-kube-api-access-wplbb\") pod \"test-curl-7355-140050664075600\" (UID: \"d679e459-1f2f-4384-b4ac-cc45d0a6b62d\") " pod="opendatahub/test-curl-7355-140050664075600" Aug 04 16:55:03.110610 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:03.110579 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wplbb\" (UniqueName: \"kubernetes.io/projected/d679e459-1f2f-4384-b4ac-cc45d0a6b62d-kube-api-access-wplbb\") pod \"test-curl-7355-140050664075600\" (UID: \"d679e459-1f2f-4384-b4ac-cc45d0a6b62d\") " pod="opendatahub/test-curl-7355-140050664075600" Aug 04 16:55:03.217997 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:03.217934 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7355-140050664075600" Aug 04 16:55:03.558021 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:03.557912 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7355-140050664075600"] Aug 04 16:55:03.560509 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:55:03.560478 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd679e459_1f2f_4384_b4ac_cc45d0a6b62d.slice/crio-996e08fce34c8f3888421ef6c105a831f97bf12c6684dc79d20abeac3a8afc98 WatchSource:0}: Error finding container 996e08fce34c8f3888421ef6c105a831f97bf12c6684dc79d20abeac3a8afc98: Status 404 returned error can't find the container with id 996e08fce34c8f3888421ef6c105a831f97bf12c6684dc79d20abeac3a8afc98 Aug 04 16:55:03.570224 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:03.570192 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7355-140050664075600" event={"ID":"d679e459-1f2f-4384-b4ac-cc45d0a6b62d","Type":"ContainerStarted","Data":"996e08fce34c8f3888421ef6c105a831f97bf12c6684dc79d20abeac3a8afc98"} Aug 04 16:55:03.571342 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:03.571324 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7355-140050664000800" Aug 04 16:55:03.571434 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:03.571339 2573 scope.go:117] "RemoveContainer" containerID="59f1d1988feac044fd001e28d0c6216caa4caffdec1ee89a5644c8220f432020" Aug 04 16:55:04.580352 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:04.580317 2573 generic.go:356] "Generic (PLEG): container finished" podID="d679e459-1f2f-4384-b4ac-cc45d0a6b62d" containerID="4f1bca9d9f3474de0c9e7c1ad2ecea1041760921f58daae47eba953c5b428946" exitCode=0 Aug 04 16:55:04.580849 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:04.580409 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7355-140050664075600" event={"ID":"d679e459-1f2f-4384-b4ac-cc45d0a6b62d","Type":"ContainerDied","Data":"4f1bca9d9f3474de0c9e7c1ad2ecea1041760921f58daae47eba953c5b428946"} Aug 04 16:55:05.719320 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:05.719296 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7355-140050664075600" Aug 04 16:55:05.739890 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:05.739862 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7355-140050664075600_d679e459-1f2f-4384-b4ac-cc45d0a6b62d/test-curl-7355-140050664075600/0.log" Aug 04 16:55:05.764170 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:05.764140 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7355-140050664075600"] Aug 04 16:55:05.770472 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:05.770442 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7355-140050664075600"] Aug 04 16:55:05.819111 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:05.819077 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wplbb\" (UniqueName: \"kubernetes.io/projected/d679e459-1f2f-4384-b4ac-cc45d0a6b62d-kube-api-access-wplbb\") pod \"d679e459-1f2f-4384-b4ac-cc45d0a6b62d\" (UID: \"d679e459-1f2f-4384-b4ac-cc45d0a6b62d\") " Aug 04 16:55:05.821172 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:05.821140 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d679e459-1f2f-4384-b4ac-cc45d0a6b62d-kube-api-access-wplbb" (OuterVolumeSpecName: "kube-api-access-wplbb") pod "d679e459-1f2f-4384-b4ac-cc45d0a6b62d" (UID: "d679e459-1f2f-4384-b4ac-cc45d0a6b62d"). InnerVolumeSpecName "kube-api-access-wplbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:55:05.919684 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:05.919591 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wplbb\" (UniqueName: \"kubernetes.io/projected/d679e459-1f2f-4384-b4ac-cc45d0a6b62d-kube-api-access-wplbb\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:55:06.591735 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:06.591709 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="996e08fce34c8f3888421ef6c105a831f97bf12c6684dc79d20abeac3a8afc98" Aug 04 16:55:06.591735 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:06.591726 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7355-140050664075600" Aug 04 16:55:06.601010 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:55:06.600979 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7355-140050664075600\" is forbidden: User \"system:node:ip-10-0-141-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-210.ec2.internal' and this object" podUID="d679e459-1f2f-4384-b4ac-cc45d0a6b62d" pod="opendatahub/test-curl-7355-140050664075600" Aug 04 16:55:07.078781 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:07.078748 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d679e459-1f2f-4384-b4ac-cc45d0a6b62d" path="/var/lib/kubelet/pods/d679e459-1f2f-4384-b4ac-cc45d0a6b62d/volumes" Aug 04 16:55:16.639917 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:16.639877 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bbe0dc-5d7d454d7d-9bk9q"] Aug 04 16:55:16.640387 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:16.640326 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d679e459-1f2f-4384-b4ac-cc45d0a6b62d" containerName="test-curl-7355-140050664075600" Aug 04 16:55:16.640387 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:16.640344 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d679e459-1f2f-4384-b4ac-cc45d0a6b62d" containerName="test-curl-7355-140050664075600" Aug 04 16:55:16.640502 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:16.640442 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d679e459-1f2f-4384-b4ac-cc45d0a6b62d" containerName="test-curl-7355-140050664075600" Aug 04 16:55:16.645114 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:16.645092 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bbe0dc-5d7d454d7d-9bk9q" Aug 04 16:55:16.647958 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:16.647937 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-bbe0dc-serving-cert\"" Aug 04 16:55:16.659929 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:16.659902 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bbe0dc-5d7d454d7d-9bk9q"] Aug 04 16:55:16.714361 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:16.714327 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsczv\" (UniqueName: \"kubernetes.io/projected/ffd1d987-4422-4b0a-b05e-9b76890b21db-kube-api-access-gsczv\") pod \"maas-api-e2e-ipp-a-bbe0dc-5d7d454d7d-9bk9q\" (UID: \"ffd1d987-4422-4b0a-b05e-9b76890b21db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bbe0dc-5d7d454d7d-9bk9q" Aug 04 16:55:16.714524 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:16.714382 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffd1d987-4422-4b0a-b05e-9b76890b21db-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bbe0dc-5d7d454d7d-9bk9q\" (UID: \"ffd1d987-4422-4b0a-b05e-9b76890b21db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bbe0dc-5d7d454d7d-9bk9q" Aug 04 16:55:16.714524 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:16.714455 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffd1d987-4422-4b0a-b05e-9b76890b21db-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bbe0dc-5d7d454d7d-9bk9q\" (UID: \"ffd1d987-4422-4b0a-b05e-9b76890b21db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bbe0dc-5d7d454d7d-9bk9q" Aug 04 16:55:16.815255 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:16.815218 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsczv\" (UniqueName: \"kubernetes.io/projected/ffd1d987-4422-4b0a-b05e-9b76890b21db-kube-api-access-gsczv\") pod \"maas-api-e2e-ipp-a-bbe0dc-5d7d454d7d-9bk9q\" (UID: \"ffd1d987-4422-4b0a-b05e-9b76890b21db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bbe0dc-5d7d454d7d-9bk9q" Aug 04 16:55:16.815466 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:16.815266 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffd1d987-4422-4b0a-b05e-9b76890b21db-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bbe0dc-5d7d454d7d-9bk9q\" (UID: \"ffd1d987-4422-4b0a-b05e-9b76890b21db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bbe0dc-5d7d454d7d-9bk9q" Aug 04 16:55:16.815466 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:16.815292 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffd1d987-4422-4b0a-b05e-9b76890b21db-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bbe0dc-5d7d454d7d-9bk9q\" (UID: \"ffd1d987-4422-4b0a-b05e-9b76890b21db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bbe0dc-5d7d454d7d-9bk9q" Aug 04 16:55:16.815992 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:16.815966 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffd1d987-4422-4b0a-b05e-9b76890b21db-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bbe0dc-5d7d454d7d-9bk9q\" (UID: \"ffd1d987-4422-4b0a-b05e-9b76890b21db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bbe0dc-5d7d454d7d-9bk9q" Aug 04 16:55:16.817782 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:16.817763 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffd1d987-4422-4b0a-b05e-9b76890b21db-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bbe0dc-5d7d454d7d-9bk9q\" (UID: \"ffd1d987-4422-4b0a-b05e-9b76890b21db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bbe0dc-5d7d454d7d-9bk9q" Aug 04 16:55:16.824476 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:16.824456 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsczv\" (UniqueName: \"kubernetes.io/projected/ffd1d987-4422-4b0a-b05e-9b76890b21db-kube-api-access-gsczv\") pod \"maas-api-e2e-ipp-a-bbe0dc-5d7d454d7d-9bk9q\" (UID: \"ffd1d987-4422-4b0a-b05e-9b76890b21db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bbe0dc-5d7d454d7d-9bk9q" Aug 04 16:55:16.954820 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:16.954785 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bbe0dc-5d7d454d7d-9bk9q" Aug 04 16:55:17.092708 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:17.092677 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bbe0dc-5d7d454d7d-9bk9q"] Aug 04 16:55:17.095976 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:55:17.095945 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd1d987_4422_4b0a_b05e_9b76890b21db.slice/crio-bd1d8ef0d08a83f94b59cf4af21dadd9303e3b2d3ea62b9cf1e8b74ac126bfe4 WatchSource:0}: Error finding container bd1d8ef0d08a83f94b59cf4af21dadd9303e3b2d3ea62b9cf1e8b74ac126bfe4: Status 404 returned error can't find the container with id bd1d8ef0d08a83f94b59cf4af21dadd9303e3b2d3ea62b9cf1e8b74ac126bfe4 Aug 04 16:55:17.643184 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:17.643152 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bbe0dc-5d7d454d7d-9bk9q" event={"ID":"ffd1d987-4422-4b0a-b05e-9b76890b21db","Type":"ContainerStarted","Data":"bd1d8ef0d08a83f94b59cf4af21dadd9303e3b2d3ea62b9cf1e8b74ac126bfe4"} Aug 04 16:55:18.648613 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:18.648580 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bbe0dc-5d7d454d7d-9bk9q" event={"ID":"ffd1d987-4422-4b0a-b05e-9b76890b21db","Type":"ContainerStarted","Data":"0e1e29db33212d9c86c307edb7c02f189b0ea2b1ccaa95886db04a63d26ef5d0"} Aug 04 16:55:18.649004 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:18.648750 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bbe0dc-5d7d454d7d-9bk9q" Aug 04 16:55:18.666234 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:18.666185 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bbe0dc-5d7d454d7d-9bk9q" podStartSLOduration=2.15137108 podStartE2EDuration="2.66617036s" totalImagesPullingTime="514.79928ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:55:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:55:18.665097072 +0000 UTC m=+3518.082639201" watchObservedRunningTime="2026-08-04 16:55:18.66617036 +0000 UTC m=+3518.083712486" Aug 04 16:55:24.658733 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:24.658706 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bbe0dc-5d7d454d7d-9bk9q" Aug 04 16:55:45.951367 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:45.951330 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg"] Aug 04 16:55:45.955996 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:45.955973 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg" Aug 04 16:55:45.958239 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:45.958215 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-54f057-serving-cert\"" Aug 04 16:55:45.964761 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:45.964732 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg"] Aug 04 16:55:46.049947 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:46.049891 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ed72da8-0dec-4b42-ae90-ec4467a1238f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg\" (UID: \"9ed72da8-0dec-4b42-ae90-ec4467a1238f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg" Aug 04 16:55:46.049947 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:46.049954 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ed72da8-0dec-4b42-ae90-ec4467a1238f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg\" (UID: \"9ed72da8-0dec-4b42-ae90-ec4467a1238f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg" Aug 04 16:55:46.050193 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:46.050027 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5bxd\" (UniqueName: \"kubernetes.io/projected/9ed72da8-0dec-4b42-ae90-ec4467a1238f-kube-api-access-b5bxd\") pod \"maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg\" (UID: \"9ed72da8-0dec-4b42-ae90-ec4467a1238f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg" Aug 04 16:55:46.150458 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:46.150423 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5bxd\" (UniqueName: \"kubernetes.io/projected/9ed72da8-0dec-4b42-ae90-ec4467a1238f-kube-api-access-b5bxd\") pod \"maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg\" (UID: \"9ed72da8-0dec-4b42-ae90-ec4467a1238f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg" Aug 04 16:55:46.150674 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:46.150468 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ed72da8-0dec-4b42-ae90-ec4467a1238f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg\" (UID: \"9ed72da8-0dec-4b42-ae90-ec4467a1238f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg" Aug 04 16:55:46.150674 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:55:46.150583 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f057-serving-cert: secret "maas-api-e2e-ipp-b-54f057-serving-cert" not found Aug 04 16:55:46.150674 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:46.150587 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ed72da8-0dec-4b42-ae90-ec4467a1238f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg\" (UID: \"9ed72da8-0dec-4b42-ae90-ec4467a1238f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg" Aug 04 16:55:46.150674 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:55:46.150632 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ed72da8-0dec-4b42-ae90-ec4467a1238f-maas-api-tls podName:9ed72da8-0dec-4b42-ae90-ec4467a1238f nodeName:}" failed. No retries permitted until 2026-08-04 16:55:46.650616108 +0000 UTC m=+3546.068158212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9ed72da8-0dec-4b42-ae90-ec4467a1238f-maas-api-tls") pod "maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg" (UID: "9ed72da8-0dec-4b42-ae90-ec4467a1238f") : secret "maas-api-e2e-ipp-b-54f057-serving-cert" not found Aug 04 16:55:46.151215 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:46.151198 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ed72da8-0dec-4b42-ae90-ec4467a1238f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg\" (UID: \"9ed72da8-0dec-4b42-ae90-ec4467a1238f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg" Aug 04 16:55:46.159440 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:46.159413 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5bxd\" (UniqueName: \"kubernetes.io/projected/9ed72da8-0dec-4b42-ae90-ec4467a1238f-kube-api-access-b5bxd\") pod \"maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg\" (UID: \"9ed72da8-0dec-4b42-ae90-ec4467a1238f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg" Aug 04 16:55:46.653425 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:46.653393 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-54f057-cfcc85dcd-6vc9p"] Aug 04 16:55:46.655787 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:46.655755 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ed72da8-0dec-4b42-ae90-ec4467a1238f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg\" (UID: \"9ed72da8-0dec-4b42-ae90-ec4467a1238f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg" Aug 04 16:55:46.655932 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:55:46.655859 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f057-serving-cert: secret "maas-api-e2e-ipp-b-54f057-serving-cert" not found Aug 04 16:55:46.655932 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:55:46.655914 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ed72da8-0dec-4b42-ae90-ec4467a1238f-maas-api-tls podName:9ed72da8-0dec-4b42-ae90-ec4467a1238f nodeName:}" failed. No retries permitted until 2026-08-04 16:55:47.655902337 +0000 UTC m=+3547.073444440 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9ed72da8-0dec-4b42-ae90-ec4467a1238f-maas-api-tls") pod "maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg" (UID: "9ed72da8-0dec-4b42-ae90-ec4467a1238f") : secret "maas-api-e2e-ipp-b-54f057-serving-cert" not found Aug 04 16:55:46.656995 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:46.656980 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-54f057-cfcc85dcd-6vc9p" Aug 04 16:55:46.659250 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:46.659226 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-54f057\"" Aug 04 16:55:46.659377 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:46.659263 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-54f057-dockercfg-ws7hj\"" Aug 04 16:55:46.668049 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:46.668024 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-54f057-cfcc85dcd-6vc9p"] Aug 04 16:55:46.756947 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:46.756905 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4182037-98d5-420f-9865-7d98c3874cf0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-54f057-cfcc85dcd-6vc9p\" (UID: \"e4182037-98d5-420f-9865-7d98c3874cf0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-54f057-cfcc85dcd-6vc9p" Aug 04 16:55:46.757113 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:46.756972 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdjwh\" (UniqueName: \"kubernetes.io/projected/e4182037-98d5-420f-9865-7d98c3874cf0-kube-api-access-jdjwh\") pod \"payload-processing-e2e-ipp-b-54f057-cfcc85dcd-6vc9p\" (UID: \"e4182037-98d5-420f-9865-7d98c3874cf0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-54f057-cfcc85dcd-6vc9p" Aug 04 16:55:46.857852 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:46.857807 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4182037-98d5-420f-9865-7d98c3874cf0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-54f057-cfcc85dcd-6vc9p\" (UID: \"e4182037-98d5-420f-9865-7d98c3874cf0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-54f057-cfcc85dcd-6vc9p" Aug 04 16:55:46.858043 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:46.857876 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdjwh\" (UniqueName: \"kubernetes.io/projected/e4182037-98d5-420f-9865-7d98c3874cf0-kube-api-access-jdjwh\") pod \"payload-processing-e2e-ipp-b-54f057-cfcc85dcd-6vc9p\" (UID: \"e4182037-98d5-420f-9865-7d98c3874cf0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-54f057-cfcc85dcd-6vc9p" Aug 04 16:55:46.858517 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:46.858496 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4182037-98d5-420f-9865-7d98c3874cf0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-54f057-cfcc85dcd-6vc9p\" (UID: \"e4182037-98d5-420f-9865-7d98c3874cf0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-54f057-cfcc85dcd-6vc9p" Aug 04 16:55:46.870386 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:46.870358 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdjwh\" (UniqueName: \"kubernetes.io/projected/e4182037-98d5-420f-9865-7d98c3874cf0-kube-api-access-jdjwh\") pod \"payload-processing-e2e-ipp-b-54f057-cfcc85dcd-6vc9p\" (UID: \"e4182037-98d5-420f-9865-7d98c3874cf0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-54f057-cfcc85dcd-6vc9p" Aug 04 16:55:46.968906 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:46.968870 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-54f057-cfcc85dcd-6vc9p" Aug 04 16:55:47.104545 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:47.104520 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-54f057-cfcc85dcd-6vc9p"] Aug 04 16:55:47.107097 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:55:47.107057 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4182037_98d5_420f_9865_7d98c3874cf0.slice/crio-d3503df9d44dab6a905991e3c2ac8447c7caff18f9f1a0fa613a5e1ca17bc0dc WatchSource:0}: Error finding container d3503df9d44dab6a905991e3c2ac8447c7caff18f9f1a0fa613a5e1ca17bc0dc: Status 404 returned error can't find the container with id d3503df9d44dab6a905991e3c2ac8447c7caff18f9f1a0fa613a5e1ca17bc0dc Aug 04 16:55:47.664012 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:47.663977 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ed72da8-0dec-4b42-ae90-ec4467a1238f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg\" (UID: \"9ed72da8-0dec-4b42-ae90-ec4467a1238f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg" Aug 04 16:55:47.666521 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:47.666482 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ed72da8-0dec-4b42-ae90-ec4467a1238f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg\" (UID: \"9ed72da8-0dec-4b42-ae90-ec4467a1238f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg" Aug 04 16:55:47.768631 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:47.768581 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg" Aug 04 16:55:47.787222 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:47.787188 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-54f057-cfcc85dcd-6vc9p" event={"ID":"e4182037-98d5-420f-9865-7d98c3874cf0","Type":"ContainerStarted","Data":"76d717be3920c95fad7352e2a5727237e89475382cb8129a22d43020a98768b9"} Aug 04 16:55:47.787397 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:47.787249 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-54f057-cfcc85dcd-6vc9p" event={"ID":"e4182037-98d5-420f-9865-7d98c3874cf0","Type":"ContainerStarted","Data":"d3503df9d44dab6a905991e3c2ac8447c7caff18f9f1a0fa613a5e1ca17bc0dc"} Aug 04 16:55:47.787397 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:47.787340 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-54f057-cfcc85dcd-6vc9p" Aug 04 16:55:47.803714 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:47.803663 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-54f057-cfcc85dcd-6vc9p" podStartSLOduration=1.803643355 podStartE2EDuration="1.803643355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:55:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:55:47.802843873 +0000 UTC m=+3547.220386010" watchObservedRunningTime="2026-08-04 16:55:47.803643355 +0000 UTC m=+3547.221185481" Aug 04 16:55:47.902035 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:47.902001 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg"] Aug 04 16:55:47.905664 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:55:47.905629 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed72da8_0dec_4b42_ae90_ec4467a1238f.slice/crio-c6423cab0c69f412a066e2c3d8abbd832cbd273078f5156f5af649a8684fcbb5 WatchSource:0}: Error finding container c6423cab0c69f412a066e2c3d8abbd832cbd273078f5156f5af649a8684fcbb5: Status 404 returned error can't find the container with id c6423cab0c69f412a066e2c3d8abbd832cbd273078f5156f5af649a8684fcbb5 Aug 04 16:55:48.793682 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:48.793641 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg" event={"ID":"9ed72da8-0dec-4b42-ae90-ec4467a1238f","Type":"ContainerStarted","Data":"38aa5464110c75a12dce9b0e1c58c43935913e51f35b350f6b9739a818a3413a"} Aug 04 16:55:48.793682 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:48.793701 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg" event={"ID":"9ed72da8-0dec-4b42-ae90-ec4467a1238f","Type":"ContainerStarted","Data":"c6423cab0c69f412a066e2c3d8abbd832cbd273078f5156f5af649a8684fcbb5"} Aug 04 16:55:48.794232 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:48.793870 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg" Aug 04 16:55:48.813639 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:48.813586 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg" podStartSLOduration=3.47596097 podStartE2EDuration="3.813569773s" totalImagesPullingTime="337.608803ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:55:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:55:48.811877707 +0000 UTC m=+3548.229419833" watchObservedRunningTime="2026-08-04 16:55:48.813569773 +0000 UTC m=+3548.231111905" Aug 04 16:55:54.803354 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:54.803325 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg" Aug 04 16:55:58.796416 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:55:58.796387 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-54f057-cfcc85dcd-6vc9p" Aug 04 16:56:16.908836 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:56:16.908806 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-54f057-cfcc85dcd-6vc9p_e4182037-98d5-420f-9865-7d98c3874cf0/payload-processing/0.log" Aug 04 16:56:41.226211 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:56:41.226104 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-gvb4l_c16f82ab-8340-49ee-b41e-5685d4592fd2/console-operator/2.log" Aug 04 16:56:41.233019 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:56:41.230642 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mtbc5_11f9cbee-a773-420b-9873-5c8cb7679dd8/ovn-acl-logging/0.log" Aug 04 16:56:41.233019 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:56:41.232474 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-gvb4l_c16f82ab-8340-49ee-b41e-5685d4592fd2/console-operator/2.log" Aug 04 16:56:41.236329 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:56:41.236310 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mtbc5_11f9cbee-a773-420b-9873-5c8cb7679dd8/ovn-acl-logging/0.log" Aug 04 16:57:17.453472 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:17.453444 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-54f057-cfcc85dcd-6vc9p_e4182037-98d5-420f-9865-7d98c3874cf0/payload-processing/0.log" Aug 04 16:57:22.538452 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:22.538414 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4dd86f-54b7b986f5-vmldj"] Aug 04 16:57:22.544838 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:22.544807 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4dd86f-54b7b986f5-vmldj" Aug 04 16:57:22.547294 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:22.547270 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-4dd86f-serving-cert\"" Aug 04 16:57:22.550589 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:22.550544 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4dd86f-54b7b986f5-vmldj"] Aug 04 16:57:22.694965 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:22.694919 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c0cf477-5012-4af6-91e7-786fde0108cf-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-4dd86f-54b7b986f5-vmldj\" (UID: \"0c0cf477-5012-4af6-91e7-786fde0108cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4dd86f-54b7b986f5-vmldj" Aug 04 16:57:22.695176 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:22.694976 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc7nz\" (UniqueName: \"kubernetes.io/projected/0c0cf477-5012-4af6-91e7-786fde0108cf-kube-api-access-lc7nz\") pod \"maas-api-e2e-ipp-cleanup-4dd86f-54b7b986f5-vmldj\" (UID: \"0c0cf477-5012-4af6-91e7-786fde0108cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4dd86f-54b7b986f5-vmldj" Aug 04 16:57:22.695176 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:22.695035 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c0cf477-5012-4af6-91e7-786fde0108cf-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-4dd86f-54b7b986f5-vmldj\" (UID: \"0c0cf477-5012-4af6-91e7-786fde0108cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4dd86f-54b7b986f5-vmldj" Aug 04 16:57:22.795895 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:22.795798 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c0cf477-5012-4af6-91e7-786fde0108cf-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-4dd86f-54b7b986f5-vmldj\" (UID: \"0c0cf477-5012-4af6-91e7-786fde0108cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4dd86f-54b7b986f5-vmldj" Aug 04 16:57:22.795895 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:22.795883 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c0cf477-5012-4af6-91e7-786fde0108cf-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-4dd86f-54b7b986f5-vmldj\" (UID: \"0c0cf477-5012-4af6-91e7-786fde0108cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4dd86f-54b7b986f5-vmldj" Aug 04 16:57:22.796154 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:22.795908 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc7nz\" (UniqueName: \"kubernetes.io/projected/0c0cf477-5012-4af6-91e7-786fde0108cf-kube-api-access-lc7nz\") pod \"maas-api-e2e-ipp-cleanup-4dd86f-54b7b986f5-vmldj\" (UID: \"0c0cf477-5012-4af6-91e7-786fde0108cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4dd86f-54b7b986f5-vmldj" Aug 04 16:57:22.796685 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:22.796660 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c0cf477-5012-4af6-91e7-786fde0108cf-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-4dd86f-54b7b986f5-vmldj\" (UID: \"0c0cf477-5012-4af6-91e7-786fde0108cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4dd86f-54b7b986f5-vmldj" Aug 04 16:57:22.798339 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:22.798320 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c0cf477-5012-4af6-91e7-786fde0108cf-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-4dd86f-54b7b986f5-vmldj\" (UID: \"0c0cf477-5012-4af6-91e7-786fde0108cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4dd86f-54b7b986f5-vmldj" Aug 04 16:57:22.803687 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:22.803663 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc7nz\" (UniqueName: \"kubernetes.io/projected/0c0cf477-5012-4af6-91e7-786fde0108cf-kube-api-access-lc7nz\") pod \"maas-api-e2e-ipp-cleanup-4dd86f-54b7b986f5-vmldj\" (UID: \"0c0cf477-5012-4af6-91e7-786fde0108cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4dd86f-54b7b986f5-vmldj" Aug 04 16:57:22.858751 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:22.858715 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4dd86f-54b7b986f5-vmldj" Aug 04 16:57:22.994018 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:22.993988 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4dd86f-54b7b986f5-vmldj"] Aug 04 16:57:22.996141 ip-10-0-141-210 kubenswrapper[2573]: W0804 16:57:22.996110 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c0cf477_5012_4af6_91e7_786fde0108cf.slice/crio-1c8da1c27b353c3ff1badbd3b3fe29d4c4bef3d1ea4b9a0c137806d543e474a5 WatchSource:0}: Error finding container 1c8da1c27b353c3ff1badbd3b3fe29d4c4bef3d1ea4b9a0c137806d543e474a5: Status 404 returned error can't find the container with id 1c8da1c27b353c3ff1badbd3b3fe29d4c4bef3d1ea4b9a0c137806d543e474a5 Aug 04 16:57:23.232083 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:23.232043 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4dd86f-54b7b986f5-vmldj" event={"ID":"0c0cf477-5012-4af6-91e7-786fde0108cf","Type":"ContainerStarted","Data":"1c8da1c27b353c3ff1badbd3b3fe29d4c4bef3d1ea4b9a0c137806d543e474a5"} Aug 04 16:57:24.238379 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:24.238345 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4dd86f-54b7b986f5-vmldj" event={"ID":"0c0cf477-5012-4af6-91e7-786fde0108cf","Type":"ContainerStarted","Data":"acdd3515ff2d96f3f21130b43376d48ad723c79e1c83fb1b285783751a8c54d9"} Aug 04 16:57:24.238792 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:24.238449 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4dd86f-54b7b986f5-vmldj" Aug 04 16:57:24.254535 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:24.254473 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4dd86f-54b7b986f5-vmldj" podStartSLOduration=1.9006486040000001 podStartE2EDuration="2.254453203s" totalImagesPullingTime="353.804599ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:57:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:57:24.253912401 +0000 UTC m=+3643.671454558" watchObservedRunningTime="2026-08-04 16:57:24.254453203 +0000 UTC m=+3643.671995332" Aug 04 16:57:30.248214 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:30.248185 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4dd86f-54b7b986f5-vmldj" Aug 04 16:57:41.363949 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:41.363918 2573 scope.go:117] "RemoveContainer" containerID="6c663d49a801ff188e550b75262f3f70000b72c342de760e4a34eb54008ec6bc" Aug 04 16:57:53.239268 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:53.239234 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4dd86f-54b7b986f5-vmldj"] Aug 04 16:57:53.239886 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:53.239480 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4dd86f-54b7b986f5-vmldj" podUID="0c0cf477-5012-4af6-91e7-786fde0108cf" containerName="maas-api" containerID="cri-o://acdd3515ff2d96f3f21130b43376d48ad723c79e1c83fb1b285783751a8c54d9" gracePeriod=30 Aug 04 16:57:53.368626 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:53.368594 2573 generic.go:356] "Generic (PLEG): container finished" podID="0c0cf477-5012-4af6-91e7-786fde0108cf" containerID="acdd3515ff2d96f3f21130b43376d48ad723c79e1c83fb1b285783751a8c54d9" exitCode=0 Aug 04 16:57:53.368784 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:53.368646 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4dd86f-54b7b986f5-vmldj" event={"ID":"0c0cf477-5012-4af6-91e7-786fde0108cf","Type":"ContainerDied","Data":"acdd3515ff2d96f3f21130b43376d48ad723c79e1c83fb1b285783751a8c54d9"} Aug 04 16:57:53.493257 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:53.493190 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4dd86f-54b7b986f5-vmldj" Aug 04 16:57:53.557351 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:53.557308 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c0cf477-5012-4af6-91e7-786fde0108cf-maas-api-tls\") pod \"0c0cf477-5012-4af6-91e7-786fde0108cf\" (UID: \"0c0cf477-5012-4af6-91e7-786fde0108cf\") " Aug 04 16:57:53.557542 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:53.557389 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lc7nz\" (UniqueName: \"kubernetes.io/projected/0c0cf477-5012-4af6-91e7-786fde0108cf-kube-api-access-lc7nz\") pod \"0c0cf477-5012-4af6-91e7-786fde0108cf\" (UID: \"0c0cf477-5012-4af6-91e7-786fde0108cf\") " Aug 04 16:57:53.557542 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:53.557430 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c0cf477-5012-4af6-91e7-786fde0108cf-openshift-service-ca\") pod \"0c0cf477-5012-4af6-91e7-786fde0108cf\" (UID: \"0c0cf477-5012-4af6-91e7-786fde0108cf\") " Aug 04 16:57:53.557869 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:53.557838 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c0cf477-5012-4af6-91e7-786fde0108cf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0c0cf477-5012-4af6-91e7-786fde0108cf" (UID: "0c0cf477-5012-4af6-91e7-786fde0108cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:57:53.559547 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:53.559513 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c0cf477-5012-4af6-91e7-786fde0108cf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0c0cf477-5012-4af6-91e7-786fde0108cf" (UID: "0c0cf477-5012-4af6-91e7-786fde0108cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:57:53.559547 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:53.559529 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c0cf477-5012-4af6-91e7-786fde0108cf-kube-api-access-lc7nz" (OuterVolumeSpecName: "kube-api-access-lc7nz") pod "0c0cf477-5012-4af6-91e7-786fde0108cf" (UID: "0c0cf477-5012-4af6-91e7-786fde0108cf"). InnerVolumeSpecName "kube-api-access-lc7nz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:57:53.658895 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:53.658845 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c0cf477-5012-4af6-91e7-786fde0108cf-maas-api-tls\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:57:53.658895 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:53.658890 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc7nz\" (UniqueName: \"kubernetes.io/projected/0c0cf477-5012-4af6-91e7-786fde0108cf-kube-api-access-lc7nz\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:57:53.658895 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:53.658900 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c0cf477-5012-4af6-91e7-786fde0108cf-openshift-service-ca\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:57:54.374417 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:54.374391 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4dd86f-54b7b986f5-vmldj" Aug 04 16:57:54.374908 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:54.374390 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4dd86f-54b7b986f5-vmldj" event={"ID":"0c0cf477-5012-4af6-91e7-786fde0108cf","Type":"ContainerDied","Data":"1c8da1c27b353c3ff1badbd3b3fe29d4c4bef3d1ea4b9a0c137806d543e474a5"} Aug 04 16:57:54.374908 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:54.374539 2573 scope.go:117] "RemoveContainer" containerID="acdd3515ff2d96f3f21130b43376d48ad723c79e1c83fb1b285783751a8c54d9" Aug 04 16:57:54.395345 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:54.395316 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4dd86f-54b7b986f5-vmldj"] Aug 04 16:57:54.398755 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:54.398727 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-4dd86f-54b7b986f5-vmldj"] Aug 04 16:57:55.078791 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:57:55.078759 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c0cf477-5012-4af6-91e7-786fde0108cf" path="/var/lib/kubelet/pods/0c0cf477-5012-4af6-91e7-786fde0108cf/volumes" Aug 04 16:58:19.340182 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:58:19.340146 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bbe0dc-5d7d454d7d-9bk9q"] Aug 04 16:58:19.340651 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:58:19.340394 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bbe0dc-5d7d454d7d-9bk9q" podUID="ffd1d987-4422-4b0a-b05e-9b76890b21db" containerName="maas-api" containerID="cri-o://0e1e29db33212d9c86c307edb7c02f189b0ea2b1ccaa95886db04a63d26ef5d0" gracePeriod=30 Aug 04 16:58:19.487041 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:58:19.487009 2573 generic.go:356] "Generic (PLEG): container finished" podID="ffd1d987-4422-4b0a-b05e-9b76890b21db" containerID="0e1e29db33212d9c86c307edb7c02f189b0ea2b1ccaa95886db04a63d26ef5d0" exitCode=0 Aug 04 16:58:19.487197 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:58:19.487085 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bbe0dc-5d7d454d7d-9bk9q" event={"ID":"ffd1d987-4422-4b0a-b05e-9b76890b21db","Type":"ContainerDied","Data":"0e1e29db33212d9c86c307edb7c02f189b0ea2b1ccaa95886db04a63d26ef5d0"} Aug 04 16:58:19.594127 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:58:19.594066 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bbe0dc-5d7d454d7d-9bk9q" Aug 04 16:58:19.688616 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:58:19.688578 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffd1d987-4422-4b0a-b05e-9b76890b21db-openshift-service-ca\") pod \"ffd1d987-4422-4b0a-b05e-9b76890b21db\" (UID: \"ffd1d987-4422-4b0a-b05e-9b76890b21db\") " Aug 04 16:58:19.688768 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:58:19.688622 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffd1d987-4422-4b0a-b05e-9b76890b21db-maas-api-tls\") pod \"ffd1d987-4422-4b0a-b05e-9b76890b21db\" (UID: \"ffd1d987-4422-4b0a-b05e-9b76890b21db\") " Aug 04 16:58:19.688768 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:58:19.688662 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsczv\" (UniqueName: \"kubernetes.io/projected/ffd1d987-4422-4b0a-b05e-9b76890b21db-kube-api-access-gsczv\") pod \"ffd1d987-4422-4b0a-b05e-9b76890b21db\" (UID: \"ffd1d987-4422-4b0a-b05e-9b76890b21db\") " Aug 04 16:58:19.688975 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:58:19.688952 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffd1d987-4422-4b0a-b05e-9b76890b21db-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ffd1d987-4422-4b0a-b05e-9b76890b21db" (UID: "ffd1d987-4422-4b0a-b05e-9b76890b21db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:58:19.690781 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:58:19.690761 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffd1d987-4422-4b0a-b05e-9b76890b21db-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ffd1d987-4422-4b0a-b05e-9b76890b21db" (UID: "ffd1d987-4422-4b0a-b05e-9b76890b21db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:58:19.690868 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:58:19.690836 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffd1d987-4422-4b0a-b05e-9b76890b21db-kube-api-access-gsczv" (OuterVolumeSpecName: "kube-api-access-gsczv") pod "ffd1d987-4422-4b0a-b05e-9b76890b21db" (UID: "ffd1d987-4422-4b0a-b05e-9b76890b21db"). InnerVolumeSpecName "kube-api-access-gsczv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:58:19.789962 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:58:19.789925 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsczv\" (UniqueName: \"kubernetes.io/projected/ffd1d987-4422-4b0a-b05e-9b76890b21db-kube-api-access-gsczv\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:58:19.789962 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:58:19.789955 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffd1d987-4422-4b0a-b05e-9b76890b21db-openshift-service-ca\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:58:19.789962 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:58:19.789965 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffd1d987-4422-4b0a-b05e-9b76890b21db-maas-api-tls\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:58:20.493305 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:58:20.493272 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bbe0dc-5d7d454d7d-9bk9q" event={"ID":"ffd1d987-4422-4b0a-b05e-9b76890b21db","Type":"ContainerDied","Data":"bd1d8ef0d08a83f94b59cf4af21dadd9303e3b2d3ea62b9cf1e8b74ac126bfe4"} Aug 04 16:58:20.493305 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:58:20.493303 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bbe0dc-5d7d454d7d-9bk9q" Aug 04 16:58:20.493872 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:58:20.493357 2573 scope.go:117] "RemoveContainer" containerID="0e1e29db33212d9c86c307edb7c02f189b0ea2b1ccaa95886db04a63d26ef5d0" Aug 04 16:58:20.515832 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:58:20.515805 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bbe0dc-5d7d454d7d-9bk9q"] Aug 04 16:58:20.525098 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:58:20.525064 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bbe0dc-5d7d454d7d-9bk9q"] Aug 04 16:58:21.078607 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:58:21.078576 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffd1d987-4422-4b0a-b05e-9b76890b21db" path="/var/lib/kubelet/pods/ffd1d987-4422-4b0a-b05e-9b76890b21db/volumes" Aug 04 16:59:18.799572 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:18.799527 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg"] Aug 04 16:59:18.800196 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:18.799885 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg" podUID="9ed72da8-0dec-4b42-ae90-ec4467a1238f" containerName="maas-api" containerID="cri-o://38aa5464110c75a12dce9b0e1c58c43935913e51f35b350f6b9739a818a3413a" gracePeriod=30 Aug 04 16:59:18.859512 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:18.859413 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-54f057-cfcc85dcd-6vc9p"] Aug 04 16:59:18.859718 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:18.859680 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-54f057-cfcc85dcd-6vc9p" podUID="e4182037-98d5-420f-9865-7d98c3874cf0" containerName="payload-processing" containerID="cri-o://76d717be3920c95fad7352e2a5727237e89475382cb8129a22d43020a98768b9" gracePeriod=30 Aug 04 16:59:19.112736 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:19.112711 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg" Aug 04 16:59:19.139985 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:19.139958 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-54f057-cfcc85dcd-6vc9p" Aug 04 16:59:19.203713 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:19.203678 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4182037-98d5-420f-9865-7d98c3874cf0-plugins-config-volume\") pod \"e4182037-98d5-420f-9865-7d98c3874cf0\" (UID: \"e4182037-98d5-420f-9865-7d98c3874cf0\") " Aug 04 16:59:19.203907 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:19.203734 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdjwh\" (UniqueName: \"kubernetes.io/projected/e4182037-98d5-420f-9865-7d98c3874cf0-kube-api-access-jdjwh\") pod \"e4182037-98d5-420f-9865-7d98c3874cf0\" (UID: \"e4182037-98d5-420f-9865-7d98c3874cf0\") " Aug 04 16:59:19.203907 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:19.203778 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5bxd\" (UniqueName: \"kubernetes.io/projected/9ed72da8-0dec-4b42-ae90-ec4467a1238f-kube-api-access-b5bxd\") pod \"9ed72da8-0dec-4b42-ae90-ec4467a1238f\" (UID: \"9ed72da8-0dec-4b42-ae90-ec4467a1238f\") " Aug 04 16:59:19.203907 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:19.203819 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ed72da8-0dec-4b42-ae90-ec4467a1238f-maas-api-tls\") pod \"9ed72da8-0dec-4b42-ae90-ec4467a1238f\" (UID: \"9ed72da8-0dec-4b42-ae90-ec4467a1238f\") " Aug 04 16:59:19.203907 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:19.203865 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ed72da8-0dec-4b42-ae90-ec4467a1238f-openshift-service-ca\") pod \"9ed72da8-0dec-4b42-ae90-ec4467a1238f\" (UID: \"9ed72da8-0dec-4b42-ae90-ec4467a1238f\") " Aug 04 16:59:19.204188 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:19.204162 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4182037-98d5-420f-9865-7d98c3874cf0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e4182037-98d5-420f-9865-7d98c3874cf0" (UID: "e4182037-98d5-420f-9865-7d98c3874cf0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:59:19.204332 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:19.204299 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ed72da8-0dec-4b42-ae90-ec4467a1238f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9ed72da8-0dec-4b42-ae90-ec4467a1238f" (UID: "9ed72da8-0dec-4b42-ae90-ec4467a1238f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:59:19.206019 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:19.205998 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4182037-98d5-420f-9865-7d98c3874cf0-kube-api-access-jdjwh" (OuterVolumeSpecName: "kube-api-access-jdjwh") pod "e4182037-98d5-420f-9865-7d98c3874cf0" (UID: "e4182037-98d5-420f-9865-7d98c3874cf0"). InnerVolumeSpecName "kube-api-access-jdjwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:59:19.206106 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:19.206033 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ed72da8-0dec-4b42-ae90-ec4467a1238f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9ed72da8-0dec-4b42-ae90-ec4467a1238f" (UID: "9ed72da8-0dec-4b42-ae90-ec4467a1238f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:59:19.206106 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:19.206042 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ed72da8-0dec-4b42-ae90-ec4467a1238f-kube-api-access-b5bxd" (OuterVolumeSpecName: "kube-api-access-b5bxd") pod "9ed72da8-0dec-4b42-ae90-ec4467a1238f" (UID: "9ed72da8-0dec-4b42-ae90-ec4467a1238f"). InnerVolumeSpecName "kube-api-access-b5bxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:59:19.305274 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:19.305231 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ed72da8-0dec-4b42-ae90-ec4467a1238f-maas-api-tls\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:59:19.305274 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:19.305264 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ed72da8-0dec-4b42-ae90-ec4467a1238f-openshift-service-ca\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:59:19.305274 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:19.305274 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4182037-98d5-420f-9865-7d98c3874cf0-plugins-config-volume\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:59:19.305274 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:19.305283 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdjwh\" (UniqueName: \"kubernetes.io/projected/e4182037-98d5-420f-9865-7d98c3874cf0-kube-api-access-jdjwh\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:59:19.305574 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:19.305292 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5bxd\" (UniqueName: \"kubernetes.io/projected/9ed72da8-0dec-4b42-ae90-ec4467a1238f-kube-api-access-b5bxd\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:59:19.765157 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:19.765125 2573 generic.go:356] "Generic (PLEG): container finished" podID="9ed72da8-0dec-4b42-ae90-ec4467a1238f" containerID="38aa5464110c75a12dce9b0e1c58c43935913e51f35b350f6b9739a818a3413a" exitCode=0 Aug 04 16:59:19.765368 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:19.765177 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg" Aug 04 16:59:19.765368 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:19.765218 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg" event={"ID":"9ed72da8-0dec-4b42-ae90-ec4467a1238f","Type":"ContainerDied","Data":"38aa5464110c75a12dce9b0e1c58c43935913e51f35b350f6b9739a818a3413a"} Aug 04 16:59:19.765368 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:19.765282 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg" event={"ID":"9ed72da8-0dec-4b42-ae90-ec4467a1238f","Type":"ContainerDied","Data":"c6423cab0c69f412a066e2c3d8abbd832cbd273078f5156f5af649a8684fcbb5"} Aug 04 16:59:19.765368 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:19.765314 2573 scope.go:117] "RemoveContainer" containerID="38aa5464110c75a12dce9b0e1c58c43935913e51f35b350f6b9739a818a3413a" Aug 04 16:59:19.766522 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:19.766504 2573 generic.go:356] "Generic (PLEG): container finished" podID="e4182037-98d5-420f-9865-7d98c3874cf0" containerID="76d717be3920c95fad7352e2a5727237e89475382cb8129a22d43020a98768b9" exitCode=0 Aug 04 16:59:19.766640 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:19.766563 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-54f057-cfcc85dcd-6vc9p" Aug 04 16:59:19.766704 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:19.766545 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-54f057-cfcc85dcd-6vc9p" event={"ID":"e4182037-98d5-420f-9865-7d98c3874cf0","Type":"ContainerDied","Data":"76d717be3920c95fad7352e2a5727237e89475382cb8129a22d43020a98768b9"} Aug 04 16:59:19.766704 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:19.766685 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-54f057-cfcc85dcd-6vc9p" event={"ID":"e4182037-98d5-420f-9865-7d98c3874cf0","Type":"ContainerDied","Data":"d3503df9d44dab6a905991e3c2ac8447c7caff18f9f1a0fa613a5e1ca17bc0dc"} Aug 04 16:59:19.774275 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:19.774258 2573 scope.go:117] "RemoveContainer" containerID="38aa5464110c75a12dce9b0e1c58c43935913e51f35b350f6b9739a818a3413a" Aug 04 16:59:19.774535 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:59:19.774516 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38aa5464110c75a12dce9b0e1c58c43935913e51f35b350f6b9739a818a3413a\": container with ID starting with 38aa5464110c75a12dce9b0e1c58c43935913e51f35b350f6b9739a818a3413a not found: ID does not exist" containerID="38aa5464110c75a12dce9b0e1c58c43935913e51f35b350f6b9739a818a3413a" Aug 04 16:59:19.774631 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:19.774544 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38aa5464110c75a12dce9b0e1c58c43935913e51f35b350f6b9739a818a3413a"} err="failed to get container status \"38aa5464110c75a12dce9b0e1c58c43935913e51f35b350f6b9739a818a3413a\": rpc error: code = NotFound desc = could not find container \"38aa5464110c75a12dce9b0e1c58c43935913e51f35b350f6b9739a818a3413a\": container with ID starting with 38aa5464110c75a12dce9b0e1c58c43935913e51f35b350f6b9739a818a3413a not found: ID does not exist" Aug 04 16:59:19.774631 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:19.774585 2573 scope.go:117] "RemoveContainer" containerID="76d717be3920c95fad7352e2a5727237e89475382cb8129a22d43020a98768b9" Aug 04 16:59:19.782172 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:19.782154 2573 scope.go:117] "RemoveContainer" containerID="76d717be3920c95fad7352e2a5727237e89475382cb8129a22d43020a98768b9" Aug 04 16:59:19.782411 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:59:19.782392 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76d717be3920c95fad7352e2a5727237e89475382cb8129a22d43020a98768b9\": container with ID starting with 76d717be3920c95fad7352e2a5727237e89475382cb8129a22d43020a98768b9 not found: ID does not exist" containerID="76d717be3920c95fad7352e2a5727237e89475382cb8129a22d43020a98768b9" Aug 04 16:59:19.782473 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:19.782418 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76d717be3920c95fad7352e2a5727237e89475382cb8129a22d43020a98768b9"} err="failed to get container status \"76d717be3920c95fad7352e2a5727237e89475382cb8129a22d43020a98768b9\": rpc error: code = NotFound desc = could not find container \"76d717be3920c95fad7352e2a5727237e89475382cb8129a22d43020a98768b9\": container with ID starting with 76d717be3920c95fad7352e2a5727237e89475382cb8129a22d43020a98768b9 not found: ID does not exist" Aug 04 16:59:19.788903 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:19.788882 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-54f057-cfcc85dcd-6vc9p"] Aug 04 16:59:19.793234 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:19.793215 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-54f057-cfcc85dcd-6vc9p"] Aug 04 16:59:19.803509 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:19.803488 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg"] Aug 04 16:59:19.807473 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:19.807453 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-54f057-f897f8754-5m2zg"] Aug 04 16:59:21.078456 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:21.078412 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ed72da8-0dec-4b42-ae90-ec4467a1238f" path="/var/lib/kubelet/pods/9ed72da8-0dec-4b42-ae90-ec4467a1238f/volumes" Aug 04 16:59:21.078995 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:21.078972 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4182037-98d5-420f-9865-7d98c3874cf0" path="/var/lib/kubelet/pods/e4182037-98d5-420f-9865-7d98c3874cf0/volumes" Aug 04 16:59:36.279878 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:36.279799 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-90d707-5df6477b5-rbbnn"] Aug 04 16:59:36.280386 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:36.280052 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90d707-5df6477b5-rbbnn" podUID="67a2f33c-cef6-482f-b9bb-4c39342b7e94" containerName="maas-api" containerID="cri-o://65fd8a108c65bd4481034c4fcffd20aabe939ed260c3a33fff07e21669ad6c69" gracePeriod=30 Aug 04 16:59:36.526323 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:36.526295 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90d707-5df6477b5-rbbnn" Aug 04 16:59:36.664038 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:36.663947 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67a2f33c-cef6-482f-b9bb-4c39342b7e94-maas-api-tls\") pod \"67a2f33c-cef6-482f-b9bb-4c39342b7e94\" (UID: \"67a2f33c-cef6-482f-b9bb-4c39342b7e94\") " Aug 04 16:59:36.664197 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:36.664017 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vc4n2\" (UniqueName: \"kubernetes.io/projected/67a2f33c-cef6-482f-b9bb-4c39342b7e94-kube-api-access-vc4n2\") pod \"67a2f33c-cef6-482f-b9bb-4c39342b7e94\" (UID: \"67a2f33c-cef6-482f-b9bb-4c39342b7e94\") " Aug 04 16:59:36.664197 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:36.664154 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67a2f33c-cef6-482f-b9bb-4c39342b7e94-openshift-service-ca\") pod \"67a2f33c-cef6-482f-b9bb-4c39342b7e94\" (UID: \"67a2f33c-cef6-482f-b9bb-4c39342b7e94\") " Aug 04 16:59:36.664510 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:36.664482 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67a2f33c-cef6-482f-b9bb-4c39342b7e94-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "67a2f33c-cef6-482f-b9bb-4c39342b7e94" (UID: "67a2f33c-cef6-482f-b9bb-4c39342b7e94"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:59:36.666066 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:36.666042 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a2f33c-cef6-482f-b9bb-4c39342b7e94-kube-api-access-vc4n2" (OuterVolumeSpecName: "kube-api-access-vc4n2") pod "67a2f33c-cef6-482f-b9bb-4c39342b7e94" (UID: "67a2f33c-cef6-482f-b9bb-4c39342b7e94"). InnerVolumeSpecName "kube-api-access-vc4n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:59:36.666236 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:36.666216 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67a2f33c-cef6-482f-b9bb-4c39342b7e94-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "67a2f33c-cef6-482f-b9bb-4c39342b7e94" (UID: "67a2f33c-cef6-482f-b9bb-4c39342b7e94"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:59:36.764909 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:36.764878 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67a2f33c-cef6-482f-b9bb-4c39342b7e94-maas-api-tls\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:59:36.764909 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:36.764906 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc4n2\" (UniqueName: \"kubernetes.io/projected/67a2f33c-cef6-482f-b9bb-4c39342b7e94-kube-api-access-vc4n2\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:59:36.764909 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:36.764916 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67a2f33c-cef6-482f-b9bb-4c39342b7e94-openshift-service-ca\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:59:36.850671 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:36.850636 2573 generic.go:356] "Generic (PLEG): container finished" podID="67a2f33c-cef6-482f-b9bb-4c39342b7e94" containerID="65fd8a108c65bd4481034c4fcffd20aabe939ed260c3a33fff07e21669ad6c69" exitCode=0 Aug 04 16:59:36.850847 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:36.850691 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90d707-5df6477b5-rbbnn" Aug 04 16:59:36.850847 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:36.850692 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90d707-5df6477b5-rbbnn" event={"ID":"67a2f33c-cef6-482f-b9bb-4c39342b7e94","Type":"ContainerDied","Data":"65fd8a108c65bd4481034c4fcffd20aabe939ed260c3a33fff07e21669ad6c69"} Aug 04 16:59:36.850991 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:36.850864 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-90d707-5df6477b5-rbbnn" event={"ID":"67a2f33c-cef6-482f-b9bb-4c39342b7e94","Type":"ContainerDied","Data":"a2ea11132846fb4b95b9e62abfaab7f466bfeb6471ac5059406c240c912cddf4"} Aug 04 16:59:36.850991 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:36.850903 2573 scope.go:117] "RemoveContainer" containerID="65fd8a108c65bd4481034c4fcffd20aabe939ed260c3a33fff07e21669ad6c69" Aug 04 16:59:36.860126 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:36.860113 2573 scope.go:117] "RemoveContainer" containerID="65fd8a108c65bd4481034c4fcffd20aabe939ed260c3a33fff07e21669ad6c69" Aug 04 16:59:36.860368 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:59:36.860350 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65fd8a108c65bd4481034c4fcffd20aabe939ed260c3a33fff07e21669ad6c69\": container with ID starting with 65fd8a108c65bd4481034c4fcffd20aabe939ed260c3a33fff07e21669ad6c69 not found: ID does not exist" containerID="65fd8a108c65bd4481034c4fcffd20aabe939ed260c3a33fff07e21669ad6c69" Aug 04 16:59:36.860411 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:36.860375 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65fd8a108c65bd4481034c4fcffd20aabe939ed260c3a33fff07e21669ad6c69"} err="failed to get container status \"65fd8a108c65bd4481034c4fcffd20aabe939ed260c3a33fff07e21669ad6c69\": rpc error: code = NotFound desc = could not find container \"65fd8a108c65bd4481034c4fcffd20aabe939ed260c3a33fff07e21669ad6c69\": container with ID starting with 65fd8a108c65bd4481034c4fcffd20aabe939ed260c3a33fff07e21669ad6c69 not found: ID does not exist" Aug 04 16:59:36.871739 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:36.871717 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-90d707-5df6477b5-rbbnn"] Aug 04 16:59:36.877509 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:36.877488 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-90d707-5df6477b5-rbbnn"] Aug 04 16:59:37.077844 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:37.077811 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67a2f33c-cef6-482f-b9bb-4c39342b7e94" path="/var/lib/kubelet/pods/67a2f33c-cef6-482f-b9bb-4c39342b7e94/volumes" Aug 04 16:59:48.898129 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:48.898098 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w"] Aug 04 16:59:48.898750 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:48.898321 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w" podUID="6698886a-4153-4a7d-b552-a56daafe913f" containerName="maas-api" containerID="cri-o://ea62f13f211f7f94b24a862ec9c7d1364cce6f59d638e80c53a96c4f10562b15" gracePeriod=30 Aug 04 16:59:49.142401 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:49.142375 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w" Aug 04 16:59:49.273957 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:49.273929 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6698886a-4153-4a7d-b552-a56daafe913f-maas-api-tls\") pod \"6698886a-4153-4a7d-b552-a56daafe913f\" (UID: \"6698886a-4153-4a7d-b552-a56daafe913f\") " Aug 04 16:59:49.274157 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:49.273995 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zm7j2\" (UniqueName: \"kubernetes.io/projected/6698886a-4153-4a7d-b552-a56daafe913f-kube-api-access-zm7j2\") pod \"6698886a-4153-4a7d-b552-a56daafe913f\" (UID: \"6698886a-4153-4a7d-b552-a56daafe913f\") " Aug 04 16:59:49.274157 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:49.274089 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6698886a-4153-4a7d-b552-a56daafe913f-openshift-service-ca\") pod \"6698886a-4153-4a7d-b552-a56daafe913f\" (UID: \"6698886a-4153-4a7d-b552-a56daafe913f\") " Aug 04 16:59:49.274422 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:49.274392 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6698886a-4153-4a7d-b552-a56daafe913f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6698886a-4153-4a7d-b552-a56daafe913f" (UID: "6698886a-4153-4a7d-b552-a56daafe913f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:59:49.275995 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:49.275969 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6698886a-4153-4a7d-b552-a56daafe913f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6698886a-4153-4a7d-b552-a56daafe913f" (UID: "6698886a-4153-4a7d-b552-a56daafe913f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:59:49.276114 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:49.276089 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6698886a-4153-4a7d-b552-a56daafe913f-kube-api-access-zm7j2" (OuterVolumeSpecName: "kube-api-access-zm7j2") pod "6698886a-4153-4a7d-b552-a56daafe913f" (UID: "6698886a-4153-4a7d-b552-a56daafe913f"). InnerVolumeSpecName "kube-api-access-zm7j2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:59:49.375421 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:49.375375 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm7j2\" (UniqueName: \"kubernetes.io/projected/6698886a-4153-4a7d-b552-a56daafe913f-kube-api-access-zm7j2\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:59:49.375421 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:49.375413 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6698886a-4153-4a7d-b552-a56daafe913f-openshift-service-ca\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:59:49.375421 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:49.375424 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6698886a-4153-4a7d-b552-a56daafe913f-maas-api-tls\") on node \"ip-10-0-141-210.ec2.internal\" DevicePath \"\"" Aug 04 16:59:49.922873 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:49.922835 2573 generic.go:356] "Generic (PLEG): container finished" podID="6698886a-4153-4a7d-b552-a56daafe913f" containerID="ea62f13f211f7f94b24a862ec9c7d1364cce6f59d638e80c53a96c4f10562b15" exitCode=0 Aug 04 16:59:49.923340 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:49.922905 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w" Aug 04 16:59:49.923340 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:49.922899 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w" event={"ID":"6698886a-4153-4a7d-b552-a56daafe913f","Type":"ContainerDied","Data":"ea62f13f211f7f94b24a862ec9c7d1364cce6f59d638e80c53a96c4f10562b15"} Aug 04 16:59:49.923340 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:49.923080 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w" event={"ID":"6698886a-4153-4a7d-b552-a56daafe913f","Type":"ContainerDied","Data":"1a4f0585e0335c44de14548bb53ee0b9491a7a1520778bf4dd7272ca77aa32d7"} Aug 04 16:59:49.923340 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:49.923102 2573 scope.go:117] "RemoveContainer" containerID="ea62f13f211f7f94b24a862ec9c7d1364cce6f59d638e80c53a96c4f10562b15" Aug 04 16:59:49.932708 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:49.932691 2573 scope.go:117] "RemoveContainer" containerID="ea62f13f211f7f94b24a862ec9c7d1364cce6f59d638e80c53a96c4f10562b15" Aug 04 16:59:49.932972 ip-10-0-141-210 kubenswrapper[2573]: E0804 16:59:49.932955 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea62f13f211f7f94b24a862ec9c7d1364cce6f59d638e80c53a96c4f10562b15\": container with ID starting with ea62f13f211f7f94b24a862ec9c7d1364cce6f59d638e80c53a96c4f10562b15 not found: ID does not exist" containerID="ea62f13f211f7f94b24a862ec9c7d1364cce6f59d638e80c53a96c4f10562b15" Aug 04 16:59:49.933020 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:49.932981 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea62f13f211f7f94b24a862ec9c7d1364cce6f59d638e80c53a96c4f10562b15"} err="failed to get container status \"ea62f13f211f7f94b24a862ec9c7d1364cce6f59d638e80c53a96c4f10562b15\": rpc error: code = NotFound desc = could not find container \"ea62f13f211f7f94b24a862ec9c7d1364cce6f59d638e80c53a96c4f10562b15\": container with ID starting with ea62f13f211f7f94b24a862ec9c7d1364cce6f59d638e80c53a96c4f10562b15 not found: ID does not exist" Aug 04 16:59:49.942781 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:49.942752 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w"] Aug 04 16:59:49.947208 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:49.947184 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5a3cb8-65b8fb57d9-v9f6w"] Aug 04 16:59:51.078079 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:51.078044 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6698886a-4153-4a7d-b552-a56daafe913f" path="/var/lib/kubelet/pods/6698886a-4153-4a7d-b552-a56daafe913f/volumes" Aug 04 16:59:56.668670 ip-10-0-141-210 kubenswrapper[2573]: I0804 16:59:56.668629 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b4b599967-lxh6z_cf8c329c-cf6b-4ac0-bb3f-56c9d0b78597/authorino/0.log" Aug 04 17:00:02.968706 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:02.968680 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dfb598967-6cw62_3de24931-7b30-4df7-8f31-6edef899c2a2/copy-manifests/0.log" Aug 04 17:00:02.987515 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:02.987482 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dfb598967-6cw62_3de24931-7b30-4df7-8f31-6edef899c2a2/manager/0.log" Aug 04 17:00:03.624494 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:03.624450 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-699c89f89b-794wr_6b3931da-3b90-4dbe-9651-9aa3cf4b6985/maas-api/0.log" Aug 04 17:00:05.091861 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:05.091803 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b4b599967-lxh6z_cf8c329c-cf6b-4ac0-bb3f-56c9d0b78597/authorino/0.log" Aug 04 17:00:05.436991 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:05.436915 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-qkn9q_f411eef7-205b-4846-95e4-fbf086465197/kuadrant-console-plugin/0.log" Aug 04 17:00:06.524176 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:06.524149 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78b5c5f4c4-n5rwh_472fadc0-3396-40f3-8d6d-1e93966ec942/kube-auth-proxy/0.log" Aug 04 17:00:06.770282 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:06.770251 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-8nbg7_af2ed0f3-c0b7-4dcd-83bd-90df1ca10aa4/payload-pre-processing/0.log" Aug 04 17:00:06.998287 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:06.998252 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7c65459b9d-v7hwq_820a26ee-edcd-4e18-9152-69027236d7a3/router/0.log" Aug 04 17:00:07.578376 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:07.578345 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-49b6x_a06359f6-702e-4f7f-974a-2a202ec66ecd/main/0.log" Aug 04 17:00:07.689850 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:07.689817 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm4v79_60ee5e6a-8f61-4af0-8b2e-5fa6094a694e/main/0.log" Aug 04 17:00:07.929092 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:07.929010 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-m757h_183a9d7f-d4be-4e9d-9cbd-d51762acf1c9/main/0.log" Aug 04 17:00:16.918622 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:16.918585 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-q2zlj_e05ec517-319e-4360-8a0f-6133fb238192/global-pull-secret-syncer/0.log" Aug 04 17:00:17.005669 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:17.005638 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8jnqc_975dcd50-33ff-4dbe-b7a8-43e44e042323/konnectivity-agent/0.log" Aug 04 17:00:17.119110 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:17.119073 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-210.ec2.internal_fd26de1bd1ac6d99210ec0097a02c100/haproxy/0.log" Aug 04 17:00:22.864579 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:22.864519 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b4b599967-lxh6z_cf8c329c-cf6b-4ac0-bb3f-56c9d0b78597/authorino/0.log" Aug 04 17:00:22.951692 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:22.951656 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-qkn9q_f411eef7-205b-4846-95e4-fbf086465197/kuadrant-console-plugin/0.log" Aug 04 17:00:25.068596 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:25.068563 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kjbnj_3b37689f-334d-4d13-a955-bb357b709a42/node-exporter/0.log" Aug 04 17:00:25.091331 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:25.091306 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kjbnj_3b37689f-334d-4d13-a955-bb357b709a42/kube-rbac-proxy/0.log" Aug 04 17:00:25.112133 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:25.112105 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kjbnj_3b37689f-334d-4d13-a955-bb357b709a42/init-textfile/0.log" Aug 04 17:00:27.498395 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:27.498364 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-gvb4l_c16f82ab-8340-49ee-b41e-5685d4592fd2/console-operator/2.log" Aug 04 17:00:27.511040 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:27.511014 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-gvb4l_c16f82ab-8340-49ee-b41e-5685d4592fd2/console-operator/3.log" Aug 04 17:00:28.123303 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.123265 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b2bcd/perf-node-gather-daemonset-9v5k5"] Aug 04 17:00:28.123597 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.123583 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4182037-98d5-420f-9865-7d98c3874cf0" containerName="payload-processing" Aug 04 17:00:28.123651 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.123598 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4182037-98d5-420f-9865-7d98c3874cf0" containerName="payload-processing" Aug 04 17:00:28.123651 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.123612 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ed72da8-0dec-4b42-ae90-ec4467a1238f" containerName="maas-api" Aug 04 17:00:28.123651 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.123618 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ed72da8-0dec-4b42-ae90-ec4467a1238f" containerName="maas-api" Aug 04 17:00:28.123651 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.123630 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67a2f33c-cef6-482f-b9bb-4c39342b7e94" containerName="maas-api" Aug 04 17:00:28.123651 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.123637 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="67a2f33c-cef6-482f-b9bb-4c39342b7e94" containerName="maas-api" Aug 04 17:00:28.123818 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.123652 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6698886a-4153-4a7d-b552-a56daafe913f" containerName="maas-api" Aug 04 17:00:28.123818 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.123659 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6698886a-4153-4a7d-b552-a56daafe913f" containerName="maas-api" Aug 04 17:00:28.123818 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.123669 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffd1d987-4422-4b0a-b05e-9b76890b21db" containerName="maas-api" Aug 04 17:00:28.123818 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.123674 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffd1d987-4422-4b0a-b05e-9b76890b21db" containerName="maas-api" Aug 04 17:00:28.123818 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.123681 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c0cf477-5012-4af6-91e7-786fde0108cf" containerName="maas-api" Aug 04 17:00:28.123818 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.123685 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c0cf477-5012-4af6-91e7-786fde0108cf" containerName="maas-api" Aug 04 17:00:28.123818 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.123777 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c0cf477-5012-4af6-91e7-786fde0108cf" containerName="maas-api" Aug 04 17:00:28.123818 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.123790 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4182037-98d5-420f-9865-7d98c3874cf0" containerName="payload-processing" Aug 04 17:00:28.123818 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.123799 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffd1d987-4422-4b0a-b05e-9b76890b21db" containerName="maas-api" Aug 04 17:00:28.123818 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.123808 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6698886a-4153-4a7d-b552-a56daafe913f" containerName="maas-api" Aug 04 17:00:28.123818 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.123817 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ed72da8-0dec-4b42-ae90-ec4467a1238f" containerName="maas-api" Aug 04 17:00:28.124454 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.123829 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="67a2f33c-cef6-482f-b9bb-4c39342b7e94" containerName="maas-api" Aug 04 17:00:28.126761 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.126747 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-9v5k5" Aug 04 17:00:28.128747 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.128725 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b2bcd\"/\"kube-root-ca.crt\"" Aug 04 17:00:28.129309 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.129284 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-b2bcd\"/\"default-dockercfg-5lghf\"" Aug 04 17:00:28.129309 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.129301 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b2bcd\"/\"openshift-service-ca.crt\"" Aug 04 17:00:28.134298 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.134271 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b2bcd/perf-node-gather-daemonset-9v5k5"] Aug 04 17:00:28.187247 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.187209 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsmfs\" (UniqueName: \"kubernetes.io/projected/a6a67214-c0e3-4049-87e3-e840dc2a80e7-kube-api-access-hsmfs\") pod \"perf-node-gather-daemonset-9v5k5\" (UID: \"a6a67214-c0e3-4049-87e3-e840dc2a80e7\") " pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-9v5k5" Aug 04 17:00:28.187415 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.187263 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a6a67214-c0e3-4049-87e3-e840dc2a80e7-lib-modules\") pod \"perf-node-gather-daemonset-9v5k5\" (UID: \"a6a67214-c0e3-4049-87e3-e840dc2a80e7\") " pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-9v5k5" Aug 04 17:00:28.187415 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.187293 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a6a67214-c0e3-4049-87e3-e840dc2a80e7-podres\") pod \"perf-node-gather-daemonset-9v5k5\" (UID: \"a6a67214-c0e3-4049-87e3-e840dc2a80e7\") " pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-9v5k5" Aug 04 17:00:28.187415 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.187314 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a6a67214-c0e3-4049-87e3-e840dc2a80e7-proc\") pod \"perf-node-gather-daemonset-9v5k5\" (UID: \"a6a67214-c0e3-4049-87e3-e840dc2a80e7\") " pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-9v5k5" Aug 04 17:00:28.187415 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.187335 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6a67214-c0e3-4049-87e3-e840dc2a80e7-sys\") pod \"perf-node-gather-daemonset-9v5k5\" (UID: \"a6a67214-c0e3-4049-87e3-e840dc2a80e7\") " pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-9v5k5" Aug 04 17:00:28.288643 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.288605 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a6a67214-c0e3-4049-87e3-e840dc2a80e7-lib-modules\") pod \"perf-node-gather-daemonset-9v5k5\" (UID: \"a6a67214-c0e3-4049-87e3-e840dc2a80e7\") " pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-9v5k5" Aug 04 17:00:28.288643 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.288644 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a6a67214-c0e3-4049-87e3-e840dc2a80e7-podres\") pod \"perf-node-gather-daemonset-9v5k5\" (UID: \"a6a67214-c0e3-4049-87e3-e840dc2a80e7\") " pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-9v5k5" Aug 04 17:00:28.288863 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.288669 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a6a67214-c0e3-4049-87e3-e840dc2a80e7-proc\") pod \"perf-node-gather-daemonset-9v5k5\" (UID: \"a6a67214-c0e3-4049-87e3-e840dc2a80e7\") " pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-9v5k5" Aug 04 17:00:28.288863 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.288696 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6a67214-c0e3-4049-87e3-e840dc2a80e7-sys\") pod \"perf-node-gather-daemonset-9v5k5\" (UID: \"a6a67214-c0e3-4049-87e3-e840dc2a80e7\") " pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-9v5k5" Aug 04 17:00:28.288863 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.288753 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsmfs\" (UniqueName: \"kubernetes.io/projected/a6a67214-c0e3-4049-87e3-e840dc2a80e7-kube-api-access-hsmfs\") pod \"perf-node-gather-daemonset-9v5k5\" (UID: \"a6a67214-c0e3-4049-87e3-e840dc2a80e7\") " pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-9v5k5" Aug 04 17:00:28.288863 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.288785 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a6a67214-c0e3-4049-87e3-e840dc2a80e7-podres\") pod \"perf-node-gather-daemonset-9v5k5\" (UID: \"a6a67214-c0e3-4049-87e3-e840dc2a80e7\") " pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-9v5k5" Aug 04 17:00:28.288863 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.288785 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a6a67214-c0e3-4049-87e3-e840dc2a80e7-lib-modules\") pod \"perf-node-gather-daemonset-9v5k5\" (UID: \"a6a67214-c0e3-4049-87e3-e840dc2a80e7\") " pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-9v5k5" Aug 04 17:00:28.288863 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.288787 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a6a67214-c0e3-4049-87e3-e840dc2a80e7-proc\") pod \"perf-node-gather-daemonset-9v5k5\" (UID: \"a6a67214-c0e3-4049-87e3-e840dc2a80e7\") " pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-9v5k5" Aug 04 17:00:28.288863 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.288816 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6a67214-c0e3-4049-87e3-e840dc2a80e7-sys\") pod \"perf-node-gather-daemonset-9v5k5\" (UID: \"a6a67214-c0e3-4049-87e3-e840dc2a80e7\") " pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-9v5k5" Aug 04 17:00:28.296295 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.296263 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsmfs\" (UniqueName: \"kubernetes.io/projected/a6a67214-c0e3-4049-87e3-e840dc2a80e7-kube-api-access-hsmfs\") pod \"perf-node-gather-daemonset-9v5k5\" (UID: \"a6a67214-c0e3-4049-87e3-e840dc2a80e7\") " pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-9v5k5" Aug 04 17:00:28.438578 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.438475 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-9v5k5" Aug 04 17:00:28.571500 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.571463 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b2bcd/perf-node-gather-daemonset-9v5k5"] Aug 04 17:00:28.574627 ip-10-0-141-210 kubenswrapper[2573]: W0804 17:00:28.574592 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda6a67214_c0e3_4049_87e3_e840dc2a80e7.slice/crio-e73e87412d0d58f4b02a72b9591c1efb8ed220cc65529ea1b42346b8a96a97b8 WatchSource:0}: Error finding container e73e87412d0d58f4b02a72b9591c1efb8ed220cc65529ea1b42346b8a96a97b8: Status 404 returned error can't find the container with id e73e87412d0d58f4b02a72b9591c1efb8ed220cc65529ea1b42346b8a96a97b8 Aug 04 17:00:28.821167 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:28.821138 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5b8776669-d97mf_c06a604b-4484-4ead-a88b-d64926af2a2b/volume-data-source-validator/0.log" Aug 04 17:00:29.097401 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:29.097310 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-9v5k5" event={"ID":"a6a67214-c0e3-4049-87e3-e840dc2a80e7","Type":"ContainerStarted","Data":"9897e1a2de19da53e9975a13fbd222274b699fada2d1b59d406315332ec48fd3"} Aug 04 17:00:29.097401 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:29.097366 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-9v5k5" event={"ID":"a6a67214-c0e3-4049-87e3-e840dc2a80e7","Type":"ContainerStarted","Data":"e73e87412d0d58f4b02a72b9591c1efb8ed220cc65529ea1b42346b8a96a97b8"} Aug 04 17:00:29.097632 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:29.097472 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-9v5k5" Aug 04 17:00:29.112931 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:29.112877 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-9v5k5" podStartSLOduration=1.112859664 podStartE2EDuration="1.112859664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:00:29.111258941 +0000 UTC m=+3828.528801092" watchObservedRunningTime="2026-08-04 17:00:29.112859664 +0000 UTC m=+3828.530401789" Aug 04 17:00:29.889182 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:29.889150 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jpd9h_68c83082-033c-4f0a-83ad-6291422eac03/dns/0.log" Aug 04 17:00:29.908342 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:29.908312 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jpd9h_68c83082-033c-4f0a-83ad-6291422eac03/kube-rbac-proxy/0.log" Aug 04 17:00:30.014756 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:30.014728 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zfqf9_482db4e4-6c5f-4c1e-9666-88f470fe67db/dns-node-resolver/0.log" Aug 04 17:00:30.621085 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:30.621053 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rz25w_c10e0ff5-2215-4032-976a-907e6dccb2d3/node-ca/0.log" Aug 04 17:00:31.787942 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:31.787913 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78b5c5f4c4-n5rwh_472fadc0-3396-40f3-8d6d-1e93966ec942/kube-auth-proxy/0.log" Aug 04 17:00:31.864279 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:31.864243 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-8nbg7_af2ed0f3-c0b7-4dcd-83bd-90df1ca10aa4/payload-pre-processing/0.log" Aug 04 17:00:31.925535 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:31.925505 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7c65459b9d-v7hwq_820a26ee-edcd-4e18-9152-69027236d7a3/router/0.log" Aug 04 17:00:32.731716 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:32.731682 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-q6m46_80648f1b-68e3-44d7-b0f9-8e93d70c9413/serve-healthcheck-canary/0.log" Aug 04 17:00:33.326993 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:33.326964 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hjc4r_e33e71a1-fc96-40e9-84e8-2e0ceda39242/kube-rbac-proxy/0.log" Aug 04 17:00:33.346899 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:33.346870 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hjc4r_e33e71a1-fc96-40e9-84e8-2e0ceda39242/exporter/0.log" Aug 04 17:00:33.367818 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:33.367792 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hjc4r_e33e71a1-fc96-40e9-84e8-2e0ceda39242/extractor/0.log" Aug 04 17:00:35.113797 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:35.113770 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-9v5k5" Aug 04 17:00:36.260687 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:36.260652 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dfb598967-6cw62_3de24931-7b30-4df7-8f31-6edef899c2a2/manager/0.log" Aug 04 17:00:36.286736 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:36.286709 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dfb598967-6cw62_3de24931-7b30-4df7-8f31-6edef899c2a2/copy-manifests/0.log" Aug 04 17:00:37.797079 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:37.797043 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5967759b96-fkvhc_0964e711-82e9-4d9f-a0b9-054a8429d92b/manager/0.log" Aug 04 17:00:37.803500 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:37.803475 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78b5c5f4c4-n5rwh_472fadc0-3396-40f3-8d6d-1e93966ec942/kube-auth-proxy/0.log" Aug 04 17:00:37.861972 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:37.861927 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-8nbg7_af2ed0f3-c0b7-4dcd-83bd-90df1ca10aa4/payload-pre-processing/0.log" Aug 04 17:00:37.904498 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:37.904453 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7c65459b9d-v7hwq_820a26ee-edcd-4e18-9152-69027236d7a3/router/0.log" Aug 04 17:00:39.524047 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:39.523960 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78b5c5f4c4-n5rwh_472fadc0-3396-40f3-8d6d-1e93966ec942/kube-auth-proxy/0.log" Aug 04 17:00:39.584402 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:39.584364 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-8nbg7_af2ed0f3-c0b7-4dcd-83bd-90df1ca10aa4/payload-pre-processing/0.log" Aug 04 17:00:39.633930 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:39.633864 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7c65459b9d-v7hwq_820a26ee-edcd-4e18-9152-69027236d7a3/router/0.log" Aug 04 17:00:42.836374 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:42.836336 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-zppl2_084fc17a-af5c-4dc9-9564-ed01e2588d4b/migrator/0.log" Aug 04 17:00:42.854972 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:42.854947 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-zppl2_084fc17a-af5c-4dc9-9564-ed01e2588d4b/graceful-termination/0.log" Aug 04 17:00:44.348231 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:44.348200 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgqfs_67933315-d76a-4b2c-8c92-f0249e607072/kube-multus-additional-cni-plugins/0.log" Aug 04 17:00:44.367272 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:44.367251 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgqfs_67933315-d76a-4b2c-8c92-f0249e607072/egress-router-binary-copy/0.log" Aug 04 17:00:44.385687 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:44.385657 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgqfs_67933315-d76a-4b2c-8c92-f0249e607072/cni-plugins/0.log" Aug 04 17:00:44.405180 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:44.405153 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgqfs_67933315-d76a-4b2c-8c92-f0249e607072/bond-cni-plugin/0.log" Aug 04 17:00:44.425123 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:44.425093 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgqfs_67933315-d76a-4b2c-8c92-f0249e607072/routeoverride-cni/0.log" Aug 04 17:00:44.444083 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:44.444027 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgqfs_67933315-d76a-4b2c-8c92-f0249e607072/whereabouts-cni-bincopy/0.log" Aug 04 17:00:44.462729 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:44.462702 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgqfs_67933315-d76a-4b2c-8c92-f0249e607072/whereabouts-cni/0.log" Aug 04 17:00:44.646455 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:44.646364 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jbkjx_abba07c5-995b-4a73-8a11-cde3a8326227/kube-multus/0.log" Aug 04 17:00:44.819913 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:44.819881 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rcb4z_fb454294-c3a4-4d8d-b705-95350ed95c13/network-metrics-daemon/0.log" Aug 04 17:00:44.836757 ip-10-0-141-210 kubenswrapper[2573]: I0804 17:00:44.836721 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rcb4z_fb454294-c3a4-4d8d-b705-95350ed95c13/kube-rbac-proxy/0.log"