Jul 27 20:32:19.102233 ip-10-0-137-141 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 27 20:32:19.102248 ip-10-0-137-141 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 27 20:32:19.102258 ip-10-0-137-141 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 27 20:32:19.102570 ip-10-0-137-141 systemd[1]: Failed to start Kubernetes Kubelet. Jul 27 20:32:29.158031 ip-10-0-137-141 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 27 20:32:29.158048 ip-10-0-137-141 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 8ead4ed6d32e4455a90c23d967fb95fd -- Jul 27 20:34:30.144927 ip-10-0-137-141 systemd[1]: Starting Kubernetes Kubelet... Jul 27 20:34:30.547863 ip-10-0-137-141 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 20:34:30.547863 ip-10-0-137-141 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 27 20:34:30.547863 ip-10-0-137-141 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 20:34:30.547863 ip-10-0-137-141 kubenswrapper[2570]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 27 20:34:30.547863 ip-10-0-137-141 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 20:34:30.550455 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.550364 2570 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 27 20:34:30.558865 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.558833 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 20:34:30.558865 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.558862 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 20:34:30.558865 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.558868 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 20:34:30.558865 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.558875 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 20:34:30.559145 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.558879 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 20:34:30.559145 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.558884 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 20:34:30.559145 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.558891 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 20:34:30.559145 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.558895 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 20:34:30.559145 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.558903 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 20:34:30.559145 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.558908 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 20:34:30.559145 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.558911 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 20:34:30.559145 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.558918 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 20:34:30.559145 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.558924 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 20:34:30.559145 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.558933 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 20:34:30.559145 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.558937 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 27 20:34:30.559145 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.558941 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 20:34:30.559145 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.558950 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 20:34:30.559145 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.558955 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 20:34:30.559145 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.558958 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 20:34:30.559145 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.558962 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 20:34:30.559145 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.558967 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 20:34:30.559145 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.558971 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 20:34:30.559145 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.558974 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 20:34:30.559145 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.558978 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 20:34:30.559992 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.558986 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 20:34:30.559992 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.558990 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 20:34:30.559992 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559010 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 20:34:30.559992 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559018 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 20:34:30.559992 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559023 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 20:34:30.559992 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559030 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 20:34:30.559992 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559035 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 20:34:30.559992 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559039 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 20:34:30.559992 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559046 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 20:34:30.559992 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559050 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 20:34:30.559992 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559055 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 20:34:30.559992 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559062 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 20:34:30.559992 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559067 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 20:34:30.559992 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559077 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 20:34:30.559992 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559082 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 20:34:30.559992 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559089 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 20:34:30.559992 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559094 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 20:34:30.559992 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559101 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 20:34:30.559992 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559112 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 20:34:30.560905 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559116 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 20:34:30.560905 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559124 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 20:34:30.560905 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559128 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 20:34:30.560905 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559133 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 20:34:30.560905 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559140 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 20:34:30.560905 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559144 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 20:34:30.560905 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559148 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 20:34:30.560905 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559156 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 20:34:30.560905 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559160 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 20:34:30.560905 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559165 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 20:34:30.560905 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559169 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 20:34:30.560905 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559173 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 20:34:30.560905 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559177 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 20:34:30.560905 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559188 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 20:34:30.560905 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559192 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 20:34:30.560905 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559201 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 20:34:30.560905 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559205 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 20:34:30.560905 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559212 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 20:34:30.560905 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559217 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 20:34:30.560905 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559221 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 20:34:30.561761 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559229 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 20:34:30.561761 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559233 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 20:34:30.561761 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559240 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 20:34:30.561761 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559244 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 20:34:30.561761 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559250 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 20:34:30.561761 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559254 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 20:34:30.561761 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559259 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 20:34:30.561761 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559264 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 20:34:30.561761 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559268 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 20:34:30.561761 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559277 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 20:34:30.561761 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559282 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 20:34:30.561761 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559288 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 20:34:30.561761 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559292 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 20:34:30.561761 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559297 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 20:34:30.561761 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559307 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 20:34:30.561761 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559312 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 20:34:30.561761 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559319 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 20:34:30.561761 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559323 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 20:34:30.561761 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559327 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 20:34:30.561761 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559332 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 20:34:30.562641 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559341 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 20:34:30.562641 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559346 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 20:34:30.562641 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559354 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 20:34:30.562641 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559359 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 20:34:30.562641 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559367 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 20:34:30.562641 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559371 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 20:34:30.562641 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559376 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 20:34:30.562641 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559384 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 20:34:30.562641 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559388 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 20:34:30.562641 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559395 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 20:34:30.562641 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559400 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 20:34:30.562641 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559404 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 20:34:30.562641 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559412 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 20:34:30.562641 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559416 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 20:34:30.562641 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559426 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 20:34:30.562641 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.559432 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 20:34:30.562641 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.560958 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 20:34:30.562641 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.560970 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 20:34:30.562641 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.560975 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 20:34:30.562641 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.560988 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 20:34:30.563552 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.560993 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 20:34:30.563552 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.560998 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 20:34:30.563552 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561002 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 20:34:30.563552 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561010 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 20:34:30.563552 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561015 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 20:34:30.563552 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561020 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 20:34:30.563552 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561024 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 20:34:30.563552 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561029 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 20:34:30.563552 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561033 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 20:34:30.563552 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561037 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 20:34:30.563552 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561041 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 20:34:30.563552 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561045 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 20:34:30.563552 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561050 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 20:34:30.563552 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561056 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 20:34:30.563552 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561060 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 20:34:30.563552 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561064 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 20:34:30.563552 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561068 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 20:34:30.563552 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561072 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 20:34:30.563552 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561077 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 20:34:30.564406 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561081 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 20:34:30.564406 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561085 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 20:34:30.564406 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561092 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 20:34:30.564406 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561103 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 20:34:30.564406 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561108 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 20:34:30.564406 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561113 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 20:34:30.564406 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561123 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 20:34:30.564406 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561128 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 20:34:30.564406 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561136 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 20:34:30.564406 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561141 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 20:34:30.564406 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561145 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 20:34:30.564406 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561153 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 20:34:30.564406 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561158 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 20:34:30.564406 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561162 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 20:34:30.564406 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561170 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 20:34:30.564406 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561174 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 20:34:30.564406 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561181 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 20:34:30.564406 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561185 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 20:34:30.564406 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561190 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 20:34:30.565015 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561197 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 20:34:30.565015 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561201 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 20:34:30.565015 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561206 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 20:34:30.565015 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561214 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 20:34:30.565015 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561218 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 20:34:30.565015 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561225 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 20:34:30.565015 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561229 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 20:34:30.565015 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561233 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 20:34:30.565015 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561242 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 20:34:30.565015 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561248 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 20:34:30.565015 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561257 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 20:34:30.565015 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561261 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 20:34:30.565015 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561265 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 20:34:30.565015 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561269 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 20:34:30.565015 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561277 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 20:34:30.565015 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561281 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 20:34:30.565015 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561289 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 20:34:30.565015 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561293 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 20:34:30.565015 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561297 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 20:34:30.565015 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561305 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 20:34:30.565581 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561310 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 20:34:30.565581 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561314 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 20:34:30.565581 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561322 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 20:34:30.565581 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561326 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 20:34:30.565581 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561334 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 20:34:30.565581 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561338 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 20:34:30.565581 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561342 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 20:34:30.565581 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561349 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 20:34:30.565581 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561353 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 20:34:30.565581 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561357 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 20:34:30.565581 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561366 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 20:34:30.565581 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561372 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 20:34:30.565581 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561377 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 20:34:30.565581 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561381 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 20:34:30.565581 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561385 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 20:34:30.565581 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561389 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 20:34:30.565581 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561393 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 20:34:30.565581 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561402 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 20:34:30.565581 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561406 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 20:34:30.566192 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561415 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 20:34:30.566192 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561420 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 20:34:30.566192 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561429 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 20:34:30.566192 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561433 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 27 20:34:30.566192 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561438 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 20:34:30.566192 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561445 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 20:34:30.566192 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561449 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 20:34:30.566192 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561457 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 20:34:30.566192 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561461 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 20:34:30.566192 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561466 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 20:34:30.566192 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561473 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 20:34:30.566192 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561477 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 20:34:30.566192 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561486 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 20:34:30.566192 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561490 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 20:34:30.566192 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561497 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 20:34:30.566192 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561505 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 20:34:30.566192 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561510 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 20:34:30.566192 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.561518 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 20:34:30.566192 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562342 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 20:34:30.566903 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562355 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 20:34:30.566903 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562360 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 20:34:30.566903 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562368 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 20:34:30.566903 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562373 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 20:34:30.566903 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562382 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 20:34:30.566903 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562387 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 20:34:30.566903 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562392 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 20:34:30.566903 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562399 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 20:34:30.566903 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562403 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 20:34:30.566903 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562407 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 20:34:30.566903 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562415 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 20:34:30.566903 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562419 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 20:34:30.566903 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562426 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 20:34:30.566903 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562430 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 20:34:30.566903 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562435 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 20:34:30.566903 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562443 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 20:34:30.566903 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562447 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 20:34:30.566903 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562453 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 20:34:30.566903 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562458 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 20:34:30.566903 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562462 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 20:34:30.566903 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562467 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 20:34:30.567477 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562471 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 20:34:30.567477 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562476 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 20:34:30.567477 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562480 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 20:34:30.567477 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562484 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 20:34:30.567477 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562492 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 20:34:30.567477 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562496 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 20:34:30.567477 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562503 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 20:34:30.567477 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562508 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 20:34:30.567477 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562512 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 20:34:30.567477 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562519 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 20:34:30.567477 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562523 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 20:34:30.567477 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562531 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 20:34:30.567477 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562535 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 20:34:30.567477 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562539 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 20:34:30.567477 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562546 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 27 20:34:30.567477 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562550 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 20:34:30.567477 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562554 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 20:34:30.567477 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562561 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 20:34:30.567477 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562569 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 20:34:30.567477 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562575 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 20:34:30.568102 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562580 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 20:34:30.568102 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562584 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 20:34:30.568102 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562591 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 20:34:30.568102 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562617 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 20:34:30.568102 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562622 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 20:34:30.568102 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562626 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 20:34:30.568102 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562633 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 20:34:30.568102 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562640 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 20:34:30.568102 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562647 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 20:34:30.568102 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562651 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 20:34:30.568102 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562655 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 20:34:30.568102 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562662 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 20:34:30.568102 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562667 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 20:34:30.568102 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562674 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 20:34:30.568102 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562678 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 20:34:30.568102 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562682 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 20:34:30.568102 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562691 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 20:34:30.568102 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562696 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 20:34:30.568102 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562703 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 20:34:30.568102 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562707 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 20:34:30.568715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562711 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 20:34:30.568715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562720 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 20:34:30.568715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562726 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 20:34:30.568715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562731 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 20:34:30.568715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562736 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 20:34:30.568715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562741 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 20:34:30.568715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562745 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 20:34:30.568715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562752 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 20:34:30.568715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562756 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 20:34:30.568715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562766 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 20:34:30.568715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562775 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 20:34:30.568715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562782 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 20:34:30.568715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562786 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 20:34:30.568715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562795 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 20:34:30.568715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562799 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 20:34:30.568715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562803 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 20:34:30.568715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562810 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 20:34:30.568715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562815 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 20:34:30.568715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562819 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 20:34:30.569284 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562826 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 20:34:30.569284 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562831 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 20:34:30.569284 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562838 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 20:34:30.569284 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562842 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 20:34:30.569284 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562846 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 20:34:30.569284 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562854 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 20:34:30.569284 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562860 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 20:34:30.569284 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562869 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 20:34:30.569284 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562875 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 20:34:30.569284 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562882 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 20:34:30.569284 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562886 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 20:34:30.569284 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562893 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 20:34:30.569284 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562898 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 20:34:30.569284 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562901 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 20:34:30.569284 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562908 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 20:34:30.569284 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562912 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 20:34:30.569284 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562917 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 20:34:30.569284 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.562924 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 20:34:30.569284 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563049 2570 flags.go:64] FLAG: --address="0.0.0.0" Jul 27 20:34:30.569284 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563064 2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 27 20:34:30.569856 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563079 2570 flags.go:64] FLAG: --anonymous-auth="true" Jul 27 20:34:30.569856 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563089 2570 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 27 20:34:30.569856 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563096 2570 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 27 20:34:30.569856 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563107 2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 27 20:34:30.569856 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563117 2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 27 20:34:30.569856 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563125 2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 27 20:34:30.569856 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563133 2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 27 20:34:30.569856 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563137 2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 27 20:34:30.569856 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563146 2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 27 20:34:30.569856 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563152 2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 27 20:34:30.569856 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563159 2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 27 20:34:30.569856 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563164 2570 flags.go:64] FLAG: --cgroup-root="" Jul 27 20:34:30.569856 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563169 2570 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 27 20:34:30.569856 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563173 2570 flags.go:64] FLAG: --client-ca-file="" Jul 27 20:34:30.569856 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563179 2570 flags.go:64] FLAG: --cloud-provider="external" Jul 27 20:34:30.569856 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563187 2570 flags.go:64] FLAG: --cluster-dns="[]" Jul 27 20:34:30.569856 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563193 2570 flags.go:64] FLAG: --cluster-domain="" Jul 27 20:34:30.569856 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563201 2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 27 20:34:30.569856 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563206 2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 27 20:34:30.569856 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563214 2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 27 20:34:30.569856 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563219 2570 flags.go:64] FLAG: --container-log-max-files="5" Jul 27 20:34:30.569856 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563228 2570 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 27 20:34:30.569856 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563233 2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 27 20:34:30.570485 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563238 2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 27 20:34:30.570485 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563243 2570 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 27 20:34:30.570485 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563248 2570 flags.go:64] FLAG: --contention-profiling="false" Jul 27 20:34:30.570485 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563253 2570 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 27 20:34:30.570485 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563257 2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 27 20:34:30.570485 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563263 2570 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 27 20:34:30.570485 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563268 2570 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 27 20:34:30.570485 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563277 2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 27 20:34:30.570485 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563283 2570 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 27 20:34:30.570485 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563287 2570 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 27 20:34:30.570485 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563292 2570 flags.go:64] FLAG: --enable-load-reader="false" Jul 27 20:34:30.570485 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563296 2570 flags.go:64] FLAG: --enable-server="true" Jul 27 20:34:30.570485 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563306 2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 27 20:34:30.570485 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563314 2570 flags.go:64] FLAG: --event-burst="100" Jul 27 20:34:30.570485 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563322 2570 flags.go:64] FLAG: --event-qps="50" Jul 27 20:34:30.570485 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563326 2570 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 27 20:34:30.570485 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563334 2570 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 27 20:34:30.570485 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563338 2570 flags.go:64] FLAG: --eviction-hard="" Jul 27 20:34:30.570485 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563344 2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 27 20:34:30.570485 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563351 2570 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 27 20:34:30.570485 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563356 2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 27 20:34:30.570485 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563361 2570 flags.go:64] FLAG: --eviction-soft="" Jul 27 20:34:30.570485 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563366 2570 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 27 20:34:30.570485 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563373 2570 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 27 20:34:30.570485 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563379 2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 27 20:34:30.571119 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563383 2570 flags.go:64] FLAG: --experimental-mounter-path="" Jul 27 20:34:30.571119 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563388 2570 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 27 20:34:30.571119 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563393 2570 flags.go:64] FLAG: --fail-swap-on="true" Jul 27 20:34:30.571119 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563398 2570 flags.go:64] FLAG: --feature-gates="" Jul 27 20:34:30.571119 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563411 2570 flags.go:64] FLAG: --file-check-frequency="20s" Jul 27 20:34:30.571119 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563415 2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 27 20:34:30.571119 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563421 2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 27 20:34:30.571119 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563426 2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 27 20:34:30.571119 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563431 2570 flags.go:64] FLAG: --healthz-port="10248" Jul 27 20:34:30.571119 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563436 2570 flags.go:64] FLAG: --help="false" Jul 27 20:34:30.571119 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563443 2570 flags.go:64] FLAG: --hostname-override="ip-10-0-137-141.ec2.internal" Jul 27 20:34:30.571119 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563449 2570 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 27 20:34:30.571119 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563454 2570 flags.go:64] FLAG: --http-check-frequency="20s" Jul 27 20:34:30.571119 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563458 2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 27 20:34:30.571119 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563463 2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 27 20:34:30.571119 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563472 2570 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 27 20:34:30.571119 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563477 2570 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 27 20:34:30.571119 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563485 2570 flags.go:64] FLAG: --image-service-endpoint="" Jul 27 20:34:30.571119 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563489 2570 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 27 20:34:30.571119 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563499 2570 flags.go:64] FLAG: --kube-api-burst="100" Jul 27 20:34:30.571119 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563503 2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 27 20:34:30.571119 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563511 2570 flags.go:64] FLAG: --kube-api-qps="50" Jul 27 20:34:30.571119 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563516 2570 flags.go:64] FLAG: --kube-reserved="" Jul 27 20:34:30.571119 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563521 2570 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 27 20:34:30.571854 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563528 2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 27 20:34:30.571854 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563533 2570 flags.go:64] FLAG: --kubelet-cgroups="" Jul 27 20:34:30.571854 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563538 2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 27 20:34:30.571854 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563545 2570 flags.go:64] FLAG: --lock-file="" Jul 27 20:34:30.571854 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563550 2570 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 27 20:34:30.571854 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563554 2570 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 27 20:34:30.571854 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563561 2570 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 27 20:34:30.571854 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563575 2570 flags.go:64] FLAG: --log-json-split-stream="false" Jul 27 20:34:30.571854 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563580 2570 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 27 20:34:30.571854 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563588 2570 flags.go:64] FLAG: --log-text-split-stream="false" Jul 27 20:34:30.571854 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563593 2570 flags.go:64] FLAG: --logging-format="text" Jul 27 20:34:30.571854 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563618 2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 27 20:34:30.571854 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563624 2570 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 27 20:34:30.571854 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563631 2570 flags.go:64] FLAG: --manifest-url="" Jul 27 20:34:30.571854 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563636 2570 flags.go:64] FLAG: --manifest-url-header="" Jul 27 20:34:30.571854 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563646 2570 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 27 20:34:30.571854 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563651 2570 flags.go:64] FLAG: --max-open-files="1000000" Jul 27 20:34:30.571854 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563661 2570 flags.go:64] FLAG: --max-pods="110" Jul 27 20:34:30.571854 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563666 2570 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 27 20:34:30.571854 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563674 2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 27 20:34:30.571854 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563678 2570 flags.go:64] FLAG: --memory-manager-policy="None" Jul 27 20:34:30.571854 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563686 2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 27 20:34:30.571854 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563691 2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 27 20:34:30.571854 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563695 2570 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 27 20:34:30.571854 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563700 2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 27 20:34:30.572445 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563712 2570 flags.go:64] FLAG: --node-status-max-images="50" Jul 27 20:34:30.572445 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563717 2570 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 27 20:34:30.572445 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563724 2570 flags.go:64] FLAG: --oom-score-adj="-999" Jul 27 20:34:30.572445 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563729 2570 flags.go:64] FLAG: --pod-cidr="" Jul 27 20:34:30.572445 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563733 2570 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 27 20:34:30.572445 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563742 2570 flags.go:64] FLAG: --pod-manifest-path="" Jul 27 20:34:30.572445 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563747 2570 flags.go:64] FLAG: --pod-max-pids="-1" Jul 27 20:34:30.572445 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563752 2570 flags.go:64] FLAG: --pods-per-core="0" Jul 27 20:34:30.572445 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563757 2570 flags.go:64] FLAG: --port="10250" Jul 27 20:34:30.572445 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563762 2570 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 27 20:34:30.572445 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563766 2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09ecfcb0c1184e492" Jul 27 20:34:30.572445 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563771 2570 flags.go:64] FLAG: --qos-reserved="" Jul 27 20:34:30.572445 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563776 2570 flags.go:64] FLAG: --read-only-port="10255" Jul 27 20:34:30.572445 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563781 2570 flags.go:64] FLAG: --register-node="true" Jul 27 20:34:30.572445 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563786 2570 flags.go:64] FLAG: --register-with-taints="" Jul 27 20:34:30.572445 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563794 2570 flags.go:64] FLAG: --registry-burst="10" Jul 27 20:34:30.572445 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563799 2570 flags.go:64] FLAG: --registry-qps="5" Jul 27 20:34:30.572445 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563804 2570 flags.go:64] FLAG: --reserved-cpus="" Jul 27 20:34:30.572445 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563809 2570 flags.go:64] FLAG: --reserved-memory="" Jul 27 20:34:30.572445 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563814 2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 27 20:34:30.572445 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563819 2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 27 20:34:30.572445 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563825 2570 flags.go:64] FLAG: --rotate-certificates="false" Jul 27 20:34:30.572445 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563829 2570 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 27 20:34:30.572445 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563834 2570 flags.go:64] FLAG: --runonce="false" Jul 27 20:34:30.572445 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563839 2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 27 20:34:30.573116 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563844 2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 27 20:34:30.573116 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563849 2570 flags.go:64] FLAG: --seccomp-default="false" Jul 27 20:34:30.573116 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563854 2570 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 27 20:34:30.573116 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563858 2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 27 20:34:30.573116 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563863 2570 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 27 20:34:30.573116 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563868 2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 27 20:34:30.573116 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563873 2570 flags.go:64] FLAG: --storage-driver-password="root" Jul 27 20:34:30.573116 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563878 2570 flags.go:64] FLAG: --storage-driver-secure="false" Jul 27 20:34:30.573116 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563882 2570 flags.go:64] FLAG: --storage-driver-table="stats" Jul 27 20:34:30.573116 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563890 2570 flags.go:64] FLAG: --storage-driver-user="root" Jul 27 20:34:30.573116 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563895 2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 27 20:34:30.573116 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563900 2570 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 27 20:34:30.573116 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563905 2570 flags.go:64] FLAG: --system-cgroups="" Jul 27 20:34:30.573116 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563909 2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 27 20:34:30.573116 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563918 2570 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 27 20:34:30.573116 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563923 2570 flags.go:64] FLAG: --tls-cert-file="" Jul 27 20:34:30.573116 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563927 2570 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 27 20:34:30.573116 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563934 2570 flags.go:64] FLAG: --tls-min-version="" Jul 27 20:34:30.573116 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563938 2570 flags.go:64] FLAG: --tls-private-key-file="" Jul 27 20:34:30.573116 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563943 2570 flags.go:64] FLAG: --topology-manager-policy="none" Jul 27 20:34:30.573116 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563948 2570 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 27 20:34:30.573116 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563952 2570 flags.go:64] FLAG: --topology-manager-scope="container" Jul 27 20:34:30.573116 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563959 2570 flags.go:64] FLAG: --v="2" Jul 27 20:34:30.573116 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563965 2570 flags.go:64] FLAG: --version="false" Jul 27 20:34:30.573116 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563973 2570 flags.go:64] FLAG: --vmodule="" Jul 27 20:34:30.573744 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563980 2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 27 20:34:30.573744 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.563986 2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 27 20:34:30.573744 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564195 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 20:34:30.573744 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564203 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 20:34:30.573744 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564209 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 20:34:30.573744 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564214 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 20:34:30.573744 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564218 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 20:34:30.573744 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564222 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 20:34:30.573744 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564226 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 20:34:30.573744 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564230 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 20:34:30.573744 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564234 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 20:34:30.573744 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564239 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 20:34:30.573744 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564243 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 20:34:30.573744 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564247 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 20:34:30.573744 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564252 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 20:34:30.573744 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564256 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 20:34:30.573744 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564262 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 20:34:30.573744 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564266 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 20:34:30.573744 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564270 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 20:34:30.573744 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564274 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 20:34:30.574244 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564279 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 20:34:30.574244 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564283 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 20:34:30.574244 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564289 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 20:34:30.574244 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564296 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 20:34:30.574244 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564300 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 20:34:30.574244 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564304 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 20:34:30.574244 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564310 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 20:34:30.574244 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564314 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 27 20:34:30.574244 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564318 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 20:34:30.574244 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564324 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 20:34:30.574244 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564329 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 20:34:30.574244 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564333 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 20:34:30.574244 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564338 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 20:34:30.574244 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564343 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 20:34:30.574244 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564347 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 20:34:30.574244 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564351 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 20:34:30.574244 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564355 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 20:34:30.574244 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564360 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 20:34:30.574244 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564364 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 20:34:30.574244 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564368 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 20:34:30.574759 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564372 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 20:34:30.574759 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564376 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 20:34:30.574759 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564380 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 20:34:30.574759 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564384 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 20:34:30.574759 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564388 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 20:34:30.574759 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564392 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 20:34:30.574759 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564397 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 20:34:30.574759 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564401 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 20:34:30.574759 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564407 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 20:34:30.574759 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564411 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 20:34:30.574759 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564415 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 20:34:30.574759 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564418 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 20:34:30.574759 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564422 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 20:34:30.574759 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564426 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 20:34:30.574759 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564430 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 20:34:30.574759 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564434 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 20:34:30.574759 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564440 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 20:34:30.574759 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564446 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 20:34:30.574759 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564451 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 20:34:30.575216 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564456 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 20:34:30.575216 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564461 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 20:34:30.575216 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564466 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 20:34:30.575216 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564471 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 20:34:30.575216 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564476 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 20:34:30.575216 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564481 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 20:34:30.575216 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564486 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 20:34:30.575216 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564490 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 20:34:30.575216 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564495 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 20:34:30.575216 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564499 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 20:34:30.575216 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564503 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 20:34:30.575216 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564507 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 20:34:30.575216 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564511 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 20:34:30.575216 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564515 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 20:34:30.575216 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564519 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 20:34:30.575216 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564523 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 20:34:30.575216 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564527 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 20:34:30.575216 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564531 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 20:34:30.575216 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564536 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 20:34:30.575216 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564540 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 20:34:30.575734 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564544 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 20:34:30.575734 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564549 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 20:34:30.575734 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564553 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 20:34:30.575734 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564557 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 20:34:30.575734 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564561 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 20:34:30.575734 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564568 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 20:34:30.575734 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564573 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 20:34:30.575734 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564578 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 20:34:30.575734 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564583 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 20:34:30.575734 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564587 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 20:34:30.575734 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564591 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 20:34:30.575734 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564614 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 20:34:30.575734 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564619 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 20:34:30.575734 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564624 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 20:34:30.575734 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564630 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 20:34:30.575734 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564634 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 20:34:30.575734 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564639 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 20:34:30.575734 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564643 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 20:34:30.575734 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564647 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 20:34:30.576187 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564652 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 20:34:30.576187 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564657 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 20:34:30.576187 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.564661 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 20:34:30.576187 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.565312 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 20:34:30.576187 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.573303 2570 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 27 20:34:30.576187 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.573321 2570 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 27 20:34:30.576187 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.573343 2570 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 27 20:34:30.576187 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.573354 2570 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 27 20:34:30.576187 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573418 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 20:34:30.576187 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573423 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 20:34:30.576187 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573426 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 20:34:30.576187 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573429 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 20:34:30.576187 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573433 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 20:34:30.576187 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573435 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 20:34:30.576536 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573438 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 20:34:30.576536 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573441 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 20:34:30.576536 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573444 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 20:34:30.576536 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573446 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 20:34:30.576536 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573449 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 20:34:30.576536 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573452 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 20:34:30.576536 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573454 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 20:34:30.576536 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573458 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 20:34:30.576536 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573462 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 20:34:30.576536 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573465 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 20:34:30.576536 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573468 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 20:34:30.576536 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573470 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 20:34:30.576536 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573473 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 20:34:30.576536 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573475 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 20:34:30.576536 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573478 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 20:34:30.576536 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573480 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 20:34:30.576536 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573482 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 20:34:30.576536 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573485 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 20:34:30.576536 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573487 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 20:34:30.576536 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573490 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 20:34:30.577034 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573492 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 20:34:30.577034 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573494 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 20:34:30.577034 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573497 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 20:34:30.577034 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573507 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 20:34:30.577034 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573510 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 20:34:30.577034 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573512 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 20:34:30.577034 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573515 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 20:34:30.577034 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573517 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 20:34:30.577034 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573521 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 20:34:30.577034 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573525 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 20:34:30.577034 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573528 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 20:34:30.577034 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573530 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 20:34:30.577034 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573553 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 27 20:34:30.577034 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573557 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 20:34:30.577034 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573560 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 20:34:30.577034 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573563 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 20:34:30.577034 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573566 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 20:34:30.577034 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573568 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 20:34:30.577034 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573571 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 20:34:30.577034 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573574 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 20:34:30.577566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573577 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 20:34:30.577566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573579 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 20:34:30.577566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573583 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 20:34:30.577566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573585 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 20:34:30.577566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573588 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 20:34:30.577566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573591 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 20:34:30.577566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573594 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 20:34:30.577566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573613 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 20:34:30.577566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573618 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 20:34:30.577566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573621 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 20:34:30.577566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573624 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 20:34:30.577566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573627 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 20:34:30.577566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573630 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 20:34:30.577566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573632 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 20:34:30.577566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573635 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 20:34:30.577566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573638 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 20:34:30.577566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573647 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 20:34:30.577566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573649 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 20:34:30.577566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573652 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 20:34:30.577566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573654 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 20:34:30.578090 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573657 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 20:34:30.578090 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573659 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 20:34:30.578090 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573662 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 20:34:30.578090 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573664 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 20:34:30.578090 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573666 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 20:34:30.578090 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573669 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 20:34:30.578090 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573671 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 20:34:30.578090 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573674 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 20:34:30.578090 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573676 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 20:34:30.578090 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573679 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 20:34:30.578090 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573681 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 20:34:30.578090 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573684 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 20:34:30.578090 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573687 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 20:34:30.578090 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573690 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 20:34:30.578090 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573693 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 20:34:30.578090 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573695 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 20:34:30.578090 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573698 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 20:34:30.578090 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573700 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 20:34:30.578090 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573703 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 20:34:30.578637 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573705 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 20:34:30.578637 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573708 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 20:34:30.578637 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573710 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 20:34:30.578637 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573713 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 20:34:30.578637 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573715 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 20:34:30.578637 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573718 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 20:34:30.578637 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573720 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 20:34:30.578637 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573722 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 20:34:30.578637 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573725 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 20:34:30.578637 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573727 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 20:34:30.578637 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573735 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 20:34:30.578637 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573738 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 20:34:30.578637 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573740 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 20:34:30.578637 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573742 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 20:34:30.578637 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.573748 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 20:34:30.579060 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573900 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 20:34:30.579060 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573906 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 20:34:30.579060 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573908 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 20:34:30.579060 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573911 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 20:34:30.579060 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573915 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 20:34:30.579060 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573919 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 20:34:30.579060 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573922 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 20:34:30.579060 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573925 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 20:34:30.579060 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573928 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 20:34:30.579060 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573930 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 20:34:30.579060 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573933 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 20:34:30.579060 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573936 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 20:34:30.579060 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573939 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 20:34:30.579060 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573941 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 20:34:30.579060 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573944 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 20:34:30.579060 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573946 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 20:34:30.579060 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573949 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 20:34:30.579060 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573952 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 20:34:30.579060 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573954 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 20:34:30.579060 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573957 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 20:34:30.579566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573959 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 20:34:30.579566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573962 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 20:34:30.579566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573965 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 20:34:30.579566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573969 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 20:34:30.579566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573972 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 20:34:30.579566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573975 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 20:34:30.579566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573978 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 20:34:30.579566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573981 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 20:34:30.579566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573991 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 20:34:30.579566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573993 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 20:34:30.579566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573996 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 20:34:30.579566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.573999 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 20:34:30.579566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574001 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 20:34:30.579566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574003 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 20:34:30.579566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574006 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 20:34:30.579566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574009 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 20:34:30.579566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574011 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 20:34:30.579566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574013 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 20:34:30.579566 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574016 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 20:34:30.580130 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574018 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 20:34:30.580130 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574021 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 20:34:30.580130 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574023 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 20:34:30.580130 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574026 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 20:34:30.580130 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574028 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 20:34:30.580130 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574031 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 20:34:30.580130 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574033 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 20:34:30.580130 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574036 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 20:34:30.580130 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574038 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 20:34:30.580130 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574042 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 20:34:30.580130 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574045 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 20:34:30.580130 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574048 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 20:34:30.580130 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574050 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 20:34:30.580130 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574053 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 20:34:30.580130 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574055 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 20:34:30.580130 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574058 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 20:34:30.580130 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574061 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 20:34:30.580130 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574064 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 20:34:30.580130 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574066 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 20:34:30.580130 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574069 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 20:34:30.580715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574081 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 20:34:30.580715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574085 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 20:34:30.580715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574094 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 20:34:30.580715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574096 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 20:34:30.580715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574099 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 20:34:30.580715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574101 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 20:34:30.580715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574104 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 20:34:30.580715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574106 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 20:34:30.580715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574108 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 20:34:30.580715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574111 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 20:34:30.580715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574113 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 20:34:30.580715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574116 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 20:34:30.580715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574118 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 20:34:30.580715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574121 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 20:34:30.580715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574123 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 27 20:34:30.580715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574129 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 20:34:30.580715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574131 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 20:34:30.580715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574133 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 20:34:30.580715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574136 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 20:34:30.580715 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574138 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 20:34:30.581221 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574141 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 20:34:30.581221 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574143 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 20:34:30.581221 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574146 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 20:34:30.581221 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574148 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 20:34:30.581221 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574151 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 20:34:30.581221 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574153 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 20:34:30.581221 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574156 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 20:34:30.581221 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574158 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 20:34:30.581221 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574161 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 20:34:30.581221 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574163 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 20:34:30.581221 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574166 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 20:34:30.581221 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574168 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 20:34:30.581221 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574170 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 20:34:30.581221 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574173 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 20:34:30.581221 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574175 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 20:34:30.581221 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574183 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 20:34:30.581221 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574185 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 20:34:30.581221 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574188 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 20:34:30.581221 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574190 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 20:34:30.581221 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:30.574193 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 20:34:30.581975 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.574198 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 20:34:30.581975 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.574335 2570 server.go:962] "Client rotation is on, will bootstrap in background" Jul 27 20:34:30.581975 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.577193 2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 27 20:34:30.581975 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.578465 2570 server.go:1019] "Starting client certificate rotation" Jul 27 20:34:30.581975 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.578577 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 27 20:34:30.581975 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.578646 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 27 20:34:30.581975 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.581163 2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 27 20:34:30.584283 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.584259 2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 27 20:34:30.599087 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.599063 2570 log.go:25] "Validated CRI v1 runtime API" Jul 27 20:34:30.601226 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.601210 2570 log.go:25] "Validated CRI v1 image API" Jul 27 20:34:30.604955 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.604938 2570 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 27 20:34:30.610564 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.610538 2570 fs.go:135] Filesystem UUIDs: map[5394db44-9655-46f2-b47d-d937bf1146e6:/dev/nvme0n1p3 7137650a-526a-4a64-af04-1c18c74e2240:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 27 20:34:30.610639 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.610564 2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 27 20:34:30.615825 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.615702 2570 manager.go:216] Machine: {Timestamp:2026-07-27 20:34:30.614401372 +0000 UTC m=+0.350381650 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098956 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2886baa067e8aee272855df7eb9142 SystemUUID:ec2886ba-a067-e8ae-e272-855df7eb9142 BootID:8ead4ed6-d32e-4455-a90c-23d967fb95fd Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:df:c1:3e:9b:83 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:df:c1:3e:9b:83 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:82:c0:5f:b2:b1:80 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 27 20:34:30.615825 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.615814 2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 27 20:34:30.615994 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.615903 2570 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 27 20:34:30.617731 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.617702 2570 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 27 20:34:30.617864 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.617731 2570 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-141.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 27 20:34:30.618457 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.618447 2570 topology_manager.go:138] "Creating topology manager with none policy" Jul 27 20:34:30.618525 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.618459 2570 container_manager_linux.go:309] "Creating device plugin manager" Jul 27 20:34:30.618525 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.618466 2570 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 27 20:34:30.619085 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.619075 2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 27 20:34:30.619591 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.619568 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 27 20:34:30.619886 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.619876 2570 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 27 20:34:30.621793 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.621782 2570 state_mem.go:36] "Initialized new in-memory state store" Jul 27 20:34:30.621905 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.621896 2570 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 27 20:34:30.621944 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.621927 2570 kubelet.go:486] "Attempting to sync node with API server" Jul 27 20:34:30.621944 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.621937 2570 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 27 20:34:30.621998 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.621952 2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 27 20:34:30.621998 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.621962 2570 kubelet.go:398] "Adding apiserver pod source" Jul 27 20:34:30.621998 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.621971 2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 27 20:34:30.622344 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.622333 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 27 20:34:30.622389 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.622353 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 27 20:34:30.624469 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.624453 2570 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 27 20:34:30.626284 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.626271 2570 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 27 20:34:30.626334 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.626302 2570 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 27 20:34:30.627843 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.627832 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 27 20:34:30.627882 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.627849 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 27 20:34:30.627882 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.627855 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 27 20:34:30.627882 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.627860 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 27 20:34:30.627882 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.627869 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 27 20:34:30.627882 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.627874 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 27 20:34:30.628013 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.627886 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 27 20:34:30.628013 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.627894 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 27 20:34:30.628013 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.627901 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 27 20:34:30.628013 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.627907 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 27 20:34:30.628013 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.627915 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 27 20:34:30.628013 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.627920 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 27 20:34:30.628704 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.628693 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 27 20:34:30.628736 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.628707 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 27 20:34:30.631594 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.631575 2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-141.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 27 20:34:30.631689 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:30.631630 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 27 20:34:30.631689 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:30.631662 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-141.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 27 20:34:30.632319 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.632307 2570 server.go:1268] "Started kubelet" Jul 27 20:34:30.634493 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.633826 2570 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 27 20:34:30.634493 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.634019 2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 27 20:34:30.634493 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.634103 2570 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 27 20:34:30.634482 ip-10-0-137-141 systemd[1]: Started Kubernetes Kubelet. Jul 27 20:34:30.636395 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.636375 2570 server.go:310] "Adding debug handlers to kubelet server" Jul 27 20:34:30.636487 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.636375 2570 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 27 20:34:30.637064 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.637038 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-whmf2" Jul 27 20:34:30.639201 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:30.639177 2570 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 27 20:34:30.642628 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.642592 2570 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 27 20:34:30.642709 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.642627 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 27 20:34:30.642767 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:30.642716 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-141.ec2.internal\" not found" Jul 27 20:34:30.643163 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.643133 2570 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 27 20:34:30.643163 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.643154 2570 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 27 20:34:30.643283 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.643122 2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 27 20:34:30.643326 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.643310 2570 reconstruct.go:97] "Volume reconstruction finished" Jul 27 20:34:30.643326 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.643319 2570 reconciler.go:29] "Reconciler: start to sync state" Jul 27 20:34:30.643522 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.643503 2570 factory.go:55] Registering systemd factory Jul 27 20:34:30.643655 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.643556 2570 factory.go:223] Registration of the systemd container factory successfully Jul 27 20:34:30.644378 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.644358 2570 factory.go:153] Registering CRI-O factory Jul 27 20:34:30.644378 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.644382 2570 factory.go:223] Registration of the crio container factory successfully Jul 27 20:34:30.644502 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.644431 2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 27 20:34:30.644502 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:30.643501 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-141.ec2.internal.18c63fd15694aa7b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-141.ec2.internal,UID:ip-10-0-137-141.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-141.ec2.internal,},FirstTimestamp:2026-07-27 20:34:30.632278651 +0000 UTC m=+0.368259004,LastTimestamp:2026-07-27 20:34:30.632278651 +0000 UTC m=+0.368259004,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-141.ec2.internal,}" Jul 27 20:34:30.644502 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.644456 2570 factory.go:103] Registering Raw factory Jul 27 20:34:30.644502 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.644495 2570 manager.go:1196] Started watching for new ooms in manager Jul 27 20:34:30.644999 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.644978 2570 manager.go:349] Starting recovery of all containers Jul 27 20:34:30.645543 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.645522 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-whmf2" Jul 27 20:34:30.652096 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.652074 2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 20:34:30.654540 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:30.654368 2570 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-137-141.ec2.internal\" not found" node="ip-10-0-137-141.ec2.internal" Jul 27 20:34:30.655380 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.655363 2570 manager.go:354] Recovery completed Jul 27 20:34:30.660112 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.660100 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 20:34:30.661798 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.661783 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-141.ec2.internal" event="NodeHasSufficientMemory" Jul 27 20:34:30.661870 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.661810 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-141.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 20:34:30.661870 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.661821 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-141.ec2.internal" event="NodeHasSufficientPID" Jul 27 20:34:30.662375 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.662361 2570 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 27 20:34:30.662375 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.662374 2570 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 27 20:34:30.662450 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.662390 2570 state_mem.go:36] "Initialized new in-memory state store" Jul 27 20:34:30.662450 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.662403 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 20:34:30.663096 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.663082 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-141.ec2.internal" event="NodeHasSufficientMemory" Jul 27 20:34:30.663171 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.663106 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-141.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 20:34:30.663171 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.663119 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-141.ec2.internal" event="NodeHasSufficientPID" Jul 27 20:34:30.663866 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.663854 2570 policy_none.go:49] "None policy: Start" Jul 27 20:34:30.663905 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.663871 2570 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 27 20:34:30.663905 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.663883 2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 27 20:34:30.664849 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.664840 2570 policy_none.go:47] "Start" Jul 27 20:34:30.698240 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.698219 2570 manager.go:337] "Starting Device Plugin manager" Jul 27 20:34:30.698388 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:30.698272 2570 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 27 20:34:30.698388 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.698285 2570 server.go:87] "Starting device plugin registration server" Jul 27 20:34:30.698579 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.698565 2570 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 27 20:34:30.698664 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.698581 2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 27 20:34:30.698739 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.698721 2570 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 27 20:34:30.698828 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.698810 2570 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 27 20:34:30.698828 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.698826 2570 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 27 20:34:30.699387 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:30.699370 2570 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 27 20:34:30.699470 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:30.699414 2570 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-141.ec2.internal\" not found" Jul 27 20:34:30.798908 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.798830 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 20:34:30.799957 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.799911 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-141.ec2.internal" event="NodeHasSufficientMemory" Jul 27 20:34:30.799957 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.799944 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-141.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 20:34:30.799957 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.799956 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-141.ec2.internal" event="NodeHasSufficientPID" Jul 27 20:34:30.800072 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.799991 2570 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-141.ec2.internal" Jul 27 20:34:30.805669 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.805653 2570 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-141.ec2.internal" Jul 27 20:34:30.805737 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:30.805678 2570 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-141.ec2.internal\": node \"ip-10-0-137-141.ec2.internal\" not found" Jul 27 20:34:30.818615 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:30.818573 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-141.ec2.internal\" not found" Jul 27 20:34:30.845261 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.845233 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 27 20:34:30.846543 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.846529 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 27 20:34:30.846618 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.846567 2570 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 27 20:34:30.846618 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.846589 2570 kubelet.go:2448] "Starting kubelet main sync loop" Jul 27 20:34:30.846699 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:30.846636 2570 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 27 20:34:30.848369 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.848347 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 20:34:30.918889 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:30.918840 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-141.ec2.internal\" not found" Jul 27 20:34:30.947305 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.947266 2570 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-141.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-141.ec2.internal"] Jul 27 20:34:30.947409 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.947364 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 20:34:30.949113 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.949096 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-141.ec2.internal" event="NodeHasSufficientMemory" Jul 27 20:34:30.949189 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.949126 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-141.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 20:34:30.949189 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.949138 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-141.ec2.internal" event="NodeHasSufficientPID" Jul 27 20:34:30.950397 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.950384 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 20:34:30.950519 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.950497 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-141.ec2.internal" Jul 27 20:34:30.950627 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.950529 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 20:34:30.951133 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.951117 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-141.ec2.internal" event="NodeHasSufficientMemory" Jul 27 20:34:30.951215 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.951146 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-141.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 20:34:30.951215 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.951159 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-141.ec2.internal" event="NodeHasSufficientPID" Jul 27 20:34:30.951215 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.951161 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-141.ec2.internal" event="NodeHasSufficientMemory" Jul 27 20:34:30.951215 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.951185 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-141.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 20:34:30.951215 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.951197 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-141.ec2.internal" event="NodeHasSufficientPID" Jul 27 20:34:30.953453 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.953436 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-141.ec2.internal" Jul 27 20:34:30.953536 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.953465 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 20:34:30.954170 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.954155 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-141.ec2.internal" event="NodeHasSufficientMemory" Jul 27 20:34:30.954240 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.954186 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-141.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 20:34:30.954240 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:30.954204 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-141.ec2.internal" event="NodeHasSufficientPID" Jul 27 20:34:30.980333 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:30.980313 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-141.ec2.internal\" not found" node="ip-10-0-137-141.ec2.internal" Jul 27 20:34:30.984734 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:30.984719 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-141.ec2.internal\" not found" node="ip-10-0-137-141.ec2.internal" Jul 27 20:34:31.019231 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:31.019200 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-141.ec2.internal\" not found" Jul 27 20:34:31.045980 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:31.045938 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/41b9efd1bfa2e292df38ef8f23f5d598-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-141.ec2.internal\" (UID: \"41b9efd1bfa2e292df38ef8f23f5d598\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-141.ec2.internal" Jul 27 20:34:31.045980 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:31.045985 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41b9efd1bfa2e292df38ef8f23f5d598-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-141.ec2.internal\" (UID: \"41b9efd1bfa2e292df38ef8f23f5d598\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-141.ec2.internal" Jul 27 20:34:31.046133 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:31.046003 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/27125cf5e1c5c4dd4172f89e5d84d2fd-config\") pod \"kube-apiserver-proxy-ip-10-0-137-141.ec2.internal\" (UID: \"27125cf5e1c5c4dd4172f89e5d84d2fd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-141.ec2.internal" Jul 27 20:34:31.121500 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:31.121457 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-141.ec2.internal\" not found" Jul 27 20:34:31.146807 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:31.146779 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/41b9efd1bfa2e292df38ef8f23f5d598-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-141.ec2.internal\" (UID: \"41b9efd1bfa2e292df38ef8f23f5d598\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-141.ec2.internal" Jul 27 20:34:31.146882 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:31.146812 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41b9efd1bfa2e292df38ef8f23f5d598-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-141.ec2.internal\" (UID: \"41b9efd1bfa2e292df38ef8f23f5d598\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-141.ec2.internal" Jul 27 20:34:31.146882 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:31.146833 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/27125cf5e1c5c4dd4172f89e5d84d2fd-config\") pod \"kube-apiserver-proxy-ip-10-0-137-141.ec2.internal\" (UID: \"27125cf5e1c5c4dd4172f89e5d84d2fd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-141.ec2.internal" Jul 27 20:34:31.146882 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:31.146854 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/41b9efd1bfa2e292df38ef8f23f5d598-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-141.ec2.internal\" (UID: \"41b9efd1bfa2e292df38ef8f23f5d598\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-141.ec2.internal" Jul 27 20:34:31.146882 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:31.146881 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/27125cf5e1c5c4dd4172f89e5d84d2fd-config\") pod \"kube-apiserver-proxy-ip-10-0-137-141.ec2.internal\" (UID: \"27125cf5e1c5c4dd4172f89e5d84d2fd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-141.ec2.internal" Jul 27 20:34:31.147039 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:31.146886 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41b9efd1bfa2e292df38ef8f23f5d598-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-141.ec2.internal\" (UID: \"41b9efd1bfa2e292df38ef8f23f5d598\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-141.ec2.internal" Jul 27 20:34:31.224268 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:31.224225 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-141.ec2.internal\" not found" Jul 27 20:34:31.285342 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:31.285303 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-141.ec2.internal" Jul 27 20:34:31.288366 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:31.288345 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-141.ec2.internal" Jul 27 20:34:31.326656 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:31.326573 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-141.ec2.internal\" not found" Jul 27 20:34:31.432998 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:31.432956 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-141.ec2.internal\" not found" Jul 27 20:34:31.533132 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:31.533086 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-141.ec2.internal\" not found" Jul 27 20:34:31.578770 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:31.578683 2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 27 20:34:31.579390 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:31.578878 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 27 20:34:31.579390 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:31.578889 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 27 20:34:31.633774 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:31.633745 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-141.ec2.internal\" not found" Jul 27 20:34:31.650453 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:31.650413 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 20:29:31 +0000 UTC" deadline="2028-04-23 07:05:49.987255218 +0000 UTC" Jul 27 20:34:31.650453 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:31.650443 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15250h31m18.336815203s" Jul 27 20:34:31.735651 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:31.735575 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-141.ec2.internal\" not found" Jul 27 20:34:31.790184 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:31.790155 2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 20:34:31.843934 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:31.843853 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-141.ec2.internal" Jul 27 20:34:31.853850 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:31.853819 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 27 20:34:31.856061 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:31.856044 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-141.ec2.internal" Jul 27 20:34:31.869044 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:31.869017 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 27 20:34:32.035238 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:32.035205 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41b9efd1bfa2e292df38ef8f23f5d598.slice/crio-51f85169de866009428f82bf7101af4edab5a7b7ff271fe23d9b3552bcdd29a3 WatchSource:0}: Error finding container 51f85169de866009428f82bf7101af4edab5a7b7ff271fe23d9b3552bcdd29a3: Status 404 returned error can't find the container with id 51f85169de866009428f82bf7101af4edab5a7b7ff271fe23d9b3552bcdd29a3 Jul 27 20:34:32.035506 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:32.035489 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27125cf5e1c5c4dd4172f89e5d84d2fd.slice/crio-460e11156864cfca19a35096ae2a91d7e9e09bb55ddd3b17a49509e26ba25ddc WatchSource:0}: Error finding container 460e11156864cfca19a35096ae2a91d7e9e09bb55ddd3b17a49509e26ba25ddc: Status 404 returned error can't find the container with id 460e11156864cfca19a35096ae2a91d7e9e09bb55ddd3b17a49509e26ba25ddc Jul 27 20:34:32.041959 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.041942 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 20:34:32.218574 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.218475 2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 20:34:32.626451 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.626126 2570 apiserver.go:52] "Watching apiserver" Jul 27 20:34:32.633577 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.633549 2570 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 27 20:34:32.634040 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.634012 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-22mpg","openshift-network-operator/iptables-alerter-jhjqt","openshift-ovn-kubernetes/ovnkube-node-ltxxw","kube-system/global-pull-secret-syncer-zrhx5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j899g","openshift-dns/node-resolver-2gppc","openshift-multus/multus-additional-cni-plugins-ljtzv","openshift-multus/network-metrics-daemon-tkd2d","openshift-network-diagnostics/network-check-target-j42rs","kube-system/konnectivity-agent-l6lsr","kube-system/kube-apiserver-proxy-ip-10-0-137-141.ec2.internal","openshift-cluster-node-tuning-operator/tuned-5dp7l","openshift-image-registry/node-ca-vf9q2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-141.ec2.internal"] Jul 27 20:34:32.635996 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.635968 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ljtzv" Jul 27 20:34:32.637325 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.637300 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vf9q2" Jul 27 20:34:32.638767 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.638733 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 27 20:34:32.638991 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.638840 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 27 20:34:32.638991 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.638880 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gt42k\"" Jul 27 20:34:32.638991 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.638916 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 27 20:34:32.638991 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.638738 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 27 20:34:32.638991 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.638991 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 27 20:34:32.639489 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.639465 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.639732 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.639711 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 27 20:34:32.640038 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.640017 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-66gc5\"" Jul 27 20:34:32.640038 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.640031 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 27 20:34:32.640306 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.640285 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 27 20:34:32.641447 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.641116 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrhx5" Jul 27 20:34:32.641447 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:32.641202 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrhx5" podUID="c3561079-4fcf-46be-8d0d-9a20c39be6d9" Jul 27 20:34:32.642706 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.642681 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j899g" Jul 27 20:34:32.642832 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.642761 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 27 20:34:32.643039 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.643020 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 27 20:34:32.643106 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.643047 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 27 20:34:32.643410 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.643388 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lf8k4\"" Jul 27 20:34:32.643622 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.643583 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 27 20:34:32.643985 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.643967 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 27 20:34:32.644104 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.644090 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 27 20:34:32.644291 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.644273 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 27 20:34:32.644511 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.644495 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2gppc" Jul 27 20:34:32.645622 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.645486 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 27 20:34:32.645850 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.645832 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 27 20:34:32.646829 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.646801 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 27 20:34:32.647204 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.647174 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j42rs" Jul 27 20:34:32.647281 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:32.647263 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j42rs" podUID="58bd29b9-0422-4c07-90f2-04d39cd07db5" Jul 27 20:34:32.647378 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.647358 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wmhvr\"" Jul 27 20:34:32.647641 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.647620 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xhggx\"" Jul 27 20:34:32.648328 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.648303 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 27 20:34:32.648435 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.648373 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 27 20:34:32.650663 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.650643 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.652303 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.652280 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jhjqt" Jul 27 20:34:32.653594 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.653573 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-znbkx\"" Jul 27 20:34:32.653716 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.653592 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/35f7c023-fb9f-40ea-9f5f-547b26383bb8-os-release\") pod \"multus-additional-cni-plugins-ljtzv\" (UID: \"35f7c023-fb9f-40ea-9f5f-547b26383bb8\") " pod="openshift-multus/multus-additional-cni-plugins-ljtzv" Jul 27 20:34:32.653716 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.653625 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c5f8ec4d-adb1-4938-8887-6708463e4b02-serviceca\") pod \"node-ca-vf9q2\" (UID: \"c5f8ec4d-adb1-4938-8887-6708463e4b02\") " pod="openshift-image-registry/node-ca-vf9q2" Jul 27 20:34:32.653716 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.653651 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-etc-openvswitch\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.653716 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.653709 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b956291a-cecc-490c-8d57-8c1c36575a89-sys-fs\") pod \"aws-ebs-csi-driver-node-j899g\" (UID: \"b956291a-cecc-490c-8d57-8c1c36575a89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j899g" Jul 27 20:34:32.653944 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.653738 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35f7c023-fb9f-40ea-9f5f-547b26383bb8-system-cni-dir\") pod \"multus-additional-cni-plugins-ljtzv\" (UID: \"35f7c023-fb9f-40ea-9f5f-547b26383bb8\") " pod="openshift-multus/multus-additional-cni-plugins-ljtzv" Jul 27 20:34:32.653944 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.653752 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 27 20:34:32.653944 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.653785 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5f8ec4d-adb1-4938-8887-6708463e4b02-host\") pod \"node-ca-vf9q2\" (UID: \"c5f8ec4d-adb1-4938-8887-6708463e4b02\") " pod="openshift-image-registry/node-ca-vf9q2" Jul 27 20:34:32.653944 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.653809 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-host-cni-netd\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.653944 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.653834 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62m4x\" (UniqueName: \"kubernetes.io/projected/cda0133a-f0f3-4567-bc30-24c8ea984936-kube-api-access-62m4x\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.653944 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.653859 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpflp\" (UniqueName: \"kubernetes.io/projected/35f7c023-fb9f-40ea-9f5f-547b26383bb8-kube-api-access-fpflp\") pod \"multus-additional-cni-plugins-ljtzv\" (UID: \"35f7c023-fb9f-40ea-9f5f-547b26383bb8\") " pod="openshift-multus/multus-additional-cni-plugins-ljtzv" Jul 27 20:34:32.653944 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.653886 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b956291a-cecc-490c-8d57-8c1c36575a89-device-dir\") pod \"aws-ebs-csi-driver-node-j899g\" (UID: \"b956291a-cecc-490c-8d57-8c1c36575a89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j899g" Jul 27 20:34:32.653944 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.653911 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b956291a-cecc-490c-8d57-8c1c36575a89-etc-selinux\") pod \"aws-ebs-csi-driver-node-j899g\" (UID: \"b956291a-cecc-490c-8d57-8c1c36575a89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j899g" Jul 27 20:34:32.653944 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.653935 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c3561079-4fcf-46be-8d0d-9a20c39be6d9-dbus\") pod \"global-pull-secret-syncer-zrhx5\" (UID: \"c3561079-4fcf-46be-8d0d-9a20c39be6d9\") " pod="kube-system/global-pull-secret-syncer-zrhx5" Jul 27 20:34:32.654392 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.653961 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/35f7c023-fb9f-40ea-9f5f-547b26383bb8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ljtzv\" (UID: \"35f7c023-fb9f-40ea-9f5f-547b26383bb8\") " pod="openshift-multus/multus-additional-cni-plugins-ljtzv" Jul 27 20:34:32.654392 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.653999 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-var-lib-openvswitch\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.654392 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.654023 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9kfx\" (UniqueName: \"kubernetes.io/projected/8c122d10-fe63-4972-87c0-4cf309037cc8-kube-api-access-h9kfx\") pod \"node-resolver-2gppc\" (UID: \"8c122d10-fe63-4972-87c0-4cf309037cc8\") " pod="openshift-dns/node-resolver-2gppc" Jul 27 20:34:32.654392 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.654054 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35f7c023-fb9f-40ea-9f5f-547b26383bb8-cni-binary-copy\") pod \"multus-additional-cni-plugins-ljtzv\" (UID: \"35f7c023-fb9f-40ea-9f5f-547b26383bb8\") " pod="openshift-multus/multus-additional-cni-plugins-ljtzv" Jul 27 20:34:32.654392 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.654079 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-run-ovn\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.654392 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.654103 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-host-cni-bin\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.654392 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.654149 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c3561079-4fcf-46be-8d0d-9a20c39be6d9-kubelet-config\") pod \"global-pull-secret-syncer-zrhx5\" (UID: \"c3561079-4fcf-46be-8d0d-9a20c39be6d9\") " pod="kube-system/global-pull-secret-syncer-zrhx5" Jul 27 20:34:32.654392 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.654190 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3561079-4fcf-46be-8d0d-9a20c39be6d9-original-pull-secret\") pod \"global-pull-secret-syncer-zrhx5\" (UID: \"c3561079-4fcf-46be-8d0d-9a20c39be6d9\") " pod="kube-system/global-pull-secret-syncer-zrhx5" Jul 27 20:34:32.654392 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.654259 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l6lsr" Jul 27 20:34:32.654940 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.654258 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c122d10-fe63-4972-87c0-4cf309037cc8-tmp-dir\") pod \"node-resolver-2gppc\" (UID: \"8c122d10-fe63-4972-87c0-4cf309037cc8\") " pod="openshift-dns/node-resolver-2gppc" Jul 27 20:34:32.654940 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.654479 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-host-run-netns\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.654940 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.654509 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cda0133a-f0f3-4567-bc30-24c8ea984936-ovnkube-config\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.654940 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.654538 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cda0133a-f0f3-4567-bc30-24c8ea984936-ovnkube-script-lib\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.654940 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.654568 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35f7c023-fb9f-40ea-9f5f-547b26383bb8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ljtzv\" (UID: \"35f7c023-fb9f-40ea-9f5f-547b26383bb8\") " pod="openshift-multus/multus-additional-cni-plugins-ljtzv" Jul 27 20:34:32.654940 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.654594 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-systemd-units\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.654940 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.654638 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-run-systemd\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.654940 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.654673 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-run-openvswitch\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.654940 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.654724 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.654940 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.654753 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-host-kubelet\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.654940 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.654780 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwv5j\" (UniqueName: \"kubernetes.io/projected/b956291a-cecc-490c-8d57-8c1c36575a89-kube-api-access-lwv5j\") pod \"aws-ebs-csi-driver-node-j899g\" (UID: \"b956291a-cecc-490c-8d57-8c1c36575a89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j899g" Jul 27 20:34:32.654940 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.654803 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8c122d10-fe63-4972-87c0-4cf309037cc8-hosts-file\") pod \"node-resolver-2gppc\" (UID: \"8c122d10-fe63-4972-87c0-4cf309037cc8\") " pod="openshift-dns/node-resolver-2gppc" Jul 27 20:34:32.654940 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.654844 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b956291a-cecc-490c-8d57-8c1c36575a89-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j899g\" (UID: \"b956291a-cecc-490c-8d57-8c1c36575a89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j899g" Jul 27 20:34:32.654940 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.654874 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/35f7c023-fb9f-40ea-9f5f-547b26383bb8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ljtzv\" (UID: \"35f7c023-fb9f-40ea-9f5f-547b26383bb8\") " pod="openshift-multus/multus-additional-cni-plugins-ljtzv" Jul 27 20:34:32.654940 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.654907 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-node-log\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.655717 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.654942 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-log-socket\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.655717 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.654994 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cda0133a-f0f3-4567-bc30-24c8ea984936-env-overrides\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.655717 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.655022 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhx59\" (UniqueName: \"kubernetes.io/projected/c5f8ec4d-adb1-4938-8887-6708463e4b02-kube-api-access-jhx59\") pod \"node-ca-vf9q2\" (UID: \"c5f8ec4d-adb1-4938-8887-6708463e4b02\") " pod="openshift-image-registry/node-ca-vf9q2" Jul 27 20:34:32.655717 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.655048 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-host-slash\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.655717 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.655072 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-host-run-ovn-kubernetes\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.655717 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.655099 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cda0133a-f0f3-4567-bc30-24c8ea984936-ovn-node-metrics-cert\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.655717 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.655127 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b956291a-cecc-490c-8d57-8c1c36575a89-socket-dir\") pod \"aws-ebs-csi-driver-node-j899g\" (UID: \"b956291a-cecc-490c-8d57-8c1c36575a89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j899g" Jul 27 20:34:32.655717 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.655150 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b956291a-cecc-490c-8d57-8c1c36575a89-registration-dir\") pod \"aws-ebs-csi-driver-node-j899g\" (UID: \"b956291a-cecc-490c-8d57-8c1c36575a89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j899g" Jul 27 20:34:32.655717 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.655176 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35f7c023-fb9f-40ea-9f5f-547b26383bb8-cnibin\") pod \"multus-additional-cni-plugins-ljtzv\" (UID: \"35f7c023-fb9f-40ea-9f5f-547b26383bb8\") " pod="openshift-multus/multus-additional-cni-plugins-ljtzv" Jul 27 20:34:32.657813 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.657041 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 27 20:34:32.657813 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.657055 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 27 20:34:32.657813 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.657178 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 27 20:34:32.657813 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.657239 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 27 20:34:32.657813 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.657273 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-t9pwp\"" Jul 27 20:34:32.657813 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.657298 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 27 20:34:32.657813 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.657567 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkd2d" Jul 27 20:34:32.657813 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:32.657654 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkd2d" podUID="05ef8a1e-7bd9-4436-a007-618d151dfd5a" Jul 27 20:34:32.658248 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.658165 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9w6bg\"" Jul 27 20:34:32.658248 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.658191 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 27 20:34:32.660285 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.660244 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.662959 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.662937 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 27 20:34:32.663079 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.663055 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 27 20:34:32.663189 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.663174 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xxwlh\"" Jul 27 20:34:32.691872 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.691845 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6vfp8" Jul 27 20:34:32.700725 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.700695 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6vfp8" Jul 27 20:34:32.746230 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.746196 2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 27 20:34:32.756422 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.756379 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35f7c023-fb9f-40ea-9f5f-547b26383bb8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ljtzv\" (UID: \"35f7c023-fb9f-40ea-9f5f-547b26383bb8\") " pod="openshift-multus/multus-additional-cni-plugins-ljtzv" Jul 27 20:34:32.756581 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.756437 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-run-openvswitch\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.756581 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.756472 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-cnibin\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.756581 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.756496 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e9d570a7-3260-4097-a669-b28a69d3748a-etc-sysconfig\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.756581 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.756521 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e9d570a7-3260-4097-a669-b28a69d3748a-etc-systemd\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.756581 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.756544 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9d570a7-3260-4097-a669-b28a69d3748a-sys\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.756581 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.756570 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cda0133a-f0f3-4567-bc30-24c8ea984936-ovn-node-metrics-cert\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.756879 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.756596 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6dcq\" (UniqueName: \"kubernetes.io/projected/58bd29b9-0422-4c07-90f2-04d39cd07db5-kube-api-access-m6dcq\") pod \"network-check-target-j42rs\" (UID: \"58bd29b9-0422-4c07-90f2-04d39cd07db5\") " pod="openshift-network-diagnostics/network-check-target-j42rs" Jul 27 20:34:32.756879 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.756625 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-host-run-netns\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.756879 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.756679 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwv5j\" (UniqueName: \"kubernetes.io/projected/b956291a-cecc-490c-8d57-8c1c36575a89-kube-api-access-lwv5j\") pod \"aws-ebs-csi-driver-node-j899g\" (UID: \"b956291a-cecc-490c-8d57-8c1c36575a89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j899g" Jul 27 20:34:32.756879 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.756705 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8c122d10-fe63-4972-87c0-4cf309037cc8-hosts-file\") pod \"node-resolver-2gppc\" (UID: \"8c122d10-fe63-4972-87c0-4cf309037cc8\") " pod="openshift-dns/node-resolver-2gppc" Jul 27 20:34:32.756879 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.756730 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-node-log\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.756879 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.756754 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05ef8a1e-7bd9-4436-a007-618d151dfd5a-metrics-certs\") pod \"network-metrics-daemon-tkd2d\" (UID: \"05ef8a1e-7bd9-4436-a007-618d151dfd5a\") " pod="openshift-multus/network-metrics-daemon-tkd2d" Jul 27 20:34:32.756879 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.756794 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e9d570a7-3260-4097-a669-b28a69d3748a-etc-modprobe-d\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.756879 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.756827 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f782e3d0-da2b-4b64-ac2e-c50230b51a3a-iptables-alerter-script\") pod \"iptables-alerter-jhjqt\" (UID: \"f782e3d0-da2b-4b64-ac2e-c50230b51a3a\") " pod="openshift-network-operator/iptables-alerter-jhjqt" Jul 27 20:34:32.756879 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.756836 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-run-openvswitch\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.757212 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.756947 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9d570a7-3260-4097-a669-b28a69d3748a-host\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.757212 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.756978 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b956291a-cecc-490c-8d57-8c1c36575a89-socket-dir\") pod \"aws-ebs-csi-driver-node-j899g\" (UID: \"b956291a-cecc-490c-8d57-8c1c36575a89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j899g" Jul 27 20:34:32.757212 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757007 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b956291a-cecc-490c-8d57-8c1c36575a89-registration-dir\") pod \"aws-ebs-csi-driver-node-j899g\" (UID: \"b956291a-cecc-490c-8d57-8c1c36575a89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j899g" Jul 27 20:34:32.757212 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757034 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/35f7c023-fb9f-40ea-9f5f-547b26383bb8-os-release\") pod \"multus-additional-cni-plugins-ljtzv\" (UID: \"35f7c023-fb9f-40ea-9f5f-547b26383bb8\") " pod="openshift-multus/multus-additional-cni-plugins-ljtzv" Jul 27 20:34:32.757212 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757061 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-host-cni-netd\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.757212 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757033 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35f7c023-fb9f-40ea-9f5f-547b26383bb8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ljtzv\" (UID: \"35f7c023-fb9f-40ea-9f5f-547b26383bb8\") " pod="openshift-multus/multus-additional-cni-plugins-ljtzv" Jul 27 20:34:32.757212 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757069 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-node-log\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.757212 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757121 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-hostroot\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.757212 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757133 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b956291a-cecc-490c-8d57-8c1c36575a89-registration-dir\") pod \"aws-ebs-csi-driver-node-j899g\" (UID: \"b956291a-cecc-490c-8d57-8c1c36575a89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j899g" Jul 27 20:34:32.757212 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757136 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/35f7c023-fb9f-40ea-9f5f-547b26383bb8-os-release\") pod \"multus-additional-cni-plugins-ljtzv\" (UID: \"35f7c023-fb9f-40ea-9f5f-547b26383bb8\") " pod="openshift-multus/multus-additional-cni-plugins-ljtzv" Jul 27 20:34:32.757212 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757150 2570 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 27 20:34:32.757212 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757176 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8c122d10-fe63-4972-87c0-4cf309037cc8-hosts-file\") pod \"node-resolver-2gppc\" (UID: \"8c122d10-fe63-4972-87c0-4cf309037cc8\") " pod="openshift-dns/node-resolver-2gppc" Jul 27 20:34:32.757212 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757192 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-host-cni-netd\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.757212 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757210 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m8td\" (UniqueName: \"kubernetes.io/projected/05ef8a1e-7bd9-4436-a007-618d151dfd5a-kube-api-access-5m8td\") pod \"network-metrics-daemon-tkd2d\" (UID: \"05ef8a1e-7bd9-4436-a007-618d151dfd5a\") " pod="openshift-multus/network-metrics-daemon-tkd2d" Jul 27 20:34:32.757657 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757245 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e9d570a7-3260-4097-a669-b28a69d3748a-etc-sysctl-d\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.757657 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757249 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b956291a-cecc-490c-8d57-8c1c36575a89-socket-dir\") pod \"aws-ebs-csi-driver-node-j899g\" (UID: \"b956291a-cecc-490c-8d57-8c1c36575a89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j899g" Jul 27 20:34:32.757657 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757277 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35f7c023-fb9f-40ea-9f5f-547b26383bb8-system-cni-dir\") pod \"multus-additional-cni-plugins-ljtzv\" (UID: \"35f7c023-fb9f-40ea-9f5f-547b26383bb8\") " pod="openshift-multus/multus-additional-cni-plugins-ljtzv" Jul 27 20:34:32.757657 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757306 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-host-var-lib-cni-multus\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.757657 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757329 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35f7c023-fb9f-40ea-9f5f-547b26383bb8-system-cni-dir\") pod \"multus-additional-cni-plugins-ljtzv\" (UID: \"35f7c023-fb9f-40ea-9f5f-547b26383bb8\") " pod="openshift-multus/multus-additional-cni-plugins-ljtzv" Jul 27 20:34:32.757657 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757356 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e9d570a7-3260-4097-a669-b28a69d3748a-etc-sysctl-conf\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.757657 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757389 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9d570a7-3260-4097-a669-b28a69d3748a-tmp\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.757657 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757434 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c3561079-4fcf-46be-8d0d-9a20c39be6d9-dbus\") pod \"global-pull-secret-syncer-zrhx5\" (UID: \"c3561079-4fcf-46be-8d0d-9a20c39be6d9\") " pod="kube-system/global-pull-secret-syncer-zrhx5" Jul 27 20:34:32.757657 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757461 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f0a531c5-371d-4024-9a40-a542d5cfdeb4-multus-daemon-config\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.757657 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757488 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-host-run-multus-certs\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.757657 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757515 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48mh6\" (UniqueName: \"kubernetes.io/projected/f782e3d0-da2b-4b64-ac2e-c50230b51a3a-kube-api-access-48mh6\") pod \"iptables-alerter-jhjqt\" (UID: \"f782e3d0-da2b-4b64-ac2e-c50230b51a3a\") " pod="openshift-network-operator/iptables-alerter-jhjqt" Jul 27 20:34:32.757657 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757542 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9kfx\" (UniqueName: \"kubernetes.io/projected/8c122d10-fe63-4972-87c0-4cf309037cc8-kube-api-access-h9kfx\") pod \"node-resolver-2gppc\" (UID: \"8c122d10-fe63-4972-87c0-4cf309037cc8\") " pod="openshift-dns/node-resolver-2gppc" Jul 27 20:34:32.757657 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757550 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c3561079-4fcf-46be-8d0d-9a20c39be6d9-dbus\") pod \"global-pull-secret-syncer-zrhx5\" (UID: \"c3561079-4fcf-46be-8d0d-9a20c39be6d9\") " pod="kube-system/global-pull-secret-syncer-zrhx5" Jul 27 20:34:32.757657 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757567 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-host-cni-bin\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.757657 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757593 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-multus-socket-dir-parent\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.757657 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757606 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-host-cni-bin\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.757657 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757617 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-multus-conf-dir\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.758497 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757662 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-systemd-units\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.758497 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757702 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-run-systemd\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.758497 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757749 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.758497 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757815 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.758497 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757829 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-systemd-units\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.758497 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757853 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-run-systemd\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.758497 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757863 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-host-run-k8s-cni-cncf-io\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.758497 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757900 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9d570a7-3260-4097-a669-b28a69d3748a-lib-modules\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.758497 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757926 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-multus-cni-dir\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.758497 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757953 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-host-kubelet\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.758497 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.757979 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b956291a-cecc-490c-8d57-8c1c36575a89-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j899g\" (UID: \"b956291a-cecc-490c-8d57-8c1c36575a89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j899g" Jul 27 20:34:32.758497 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758007 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/35f7c023-fb9f-40ea-9f5f-547b26383bb8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ljtzv\" (UID: \"35f7c023-fb9f-40ea-9f5f-547b26383bb8\") " pod="openshift-multus/multus-additional-cni-plugins-ljtzv" Jul 27 20:34:32.758497 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758017 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-host-kubelet\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.758497 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758032 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-log-socket\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.758497 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758056 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cda0133a-f0f3-4567-bc30-24c8ea984936-env-overrides\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.758497 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758082 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c33a8970-0e52-4e5e-82bf-e8823e1f88d9-agent-certs\") pod \"konnectivity-agent-l6lsr\" (UID: \"c33a8970-0e52-4e5e-82bf-e8823e1f88d9\") " pod="kube-system/konnectivity-agent-l6lsr" Jul 27 20:34:32.758497 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758088 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b956291a-cecc-490c-8d57-8c1c36575a89-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j899g\" (UID: \"b956291a-cecc-490c-8d57-8c1c36575a89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j899g" Jul 27 20:34:32.759379 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758105 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9d570a7-3260-4097-a669-b28a69d3748a-etc-kubernetes\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.759379 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758133 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhx59\" (UniqueName: \"kubernetes.io/projected/c5f8ec4d-adb1-4938-8887-6708463e4b02-kube-api-access-jhx59\") pod \"node-ca-vf9q2\" (UID: \"c5f8ec4d-adb1-4938-8887-6708463e4b02\") " pod="openshift-image-registry/node-ca-vf9q2" Jul 27 20:34:32.759379 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758148 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-log-socket\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.759379 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758211 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-host-slash\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.759379 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758160 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-host-slash\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.759379 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758258 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-host-run-ovn-kubernetes\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.759379 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758287 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-os-release\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.759379 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758311 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-host-var-lib-kubelet\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.759379 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758328 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e9d570a7-3260-4097-a669-b28a69d3748a-etc-tuned\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.759379 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758346 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35f7c023-fb9f-40ea-9f5f-547b26383bb8-cnibin\") pod \"multus-additional-cni-plugins-ljtzv\" (UID: \"35f7c023-fb9f-40ea-9f5f-547b26383bb8\") " pod="openshift-multus/multus-additional-cni-plugins-ljtzv" Jul 27 20:34:32.759379 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758369 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c5f8ec4d-adb1-4938-8887-6708463e4b02-serviceca\") pod \"node-ca-vf9q2\" (UID: \"c5f8ec4d-adb1-4938-8887-6708463e4b02\") " pod="openshift-image-registry/node-ca-vf9q2" Jul 27 20:34:32.759379 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758386 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-etc-openvswitch\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.759379 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758403 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62m4x\" (UniqueName: \"kubernetes.io/projected/cda0133a-f0f3-4567-bc30-24c8ea984936-kube-api-access-62m4x\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.759379 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758418 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-etc-kubernetes\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.759379 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758435 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c33a8970-0e52-4e5e-82bf-e8823e1f88d9-konnectivity-ca\") pod \"konnectivity-agent-l6lsr\" (UID: \"c33a8970-0e52-4e5e-82bf-e8823e1f88d9\") " pod="kube-system/konnectivity-agent-l6lsr" Jul 27 20:34:32.759379 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758451 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-host-run-ovn-kubernetes\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.759379 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758461 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b956291a-cecc-490c-8d57-8c1c36575a89-sys-fs\") pod \"aws-ebs-csi-driver-node-j899g\" (UID: \"b956291a-cecc-490c-8d57-8c1c36575a89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j899g" Jul 27 20:34:32.760242 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758493 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5f8ec4d-adb1-4938-8887-6708463e4b02-host\") pod \"node-ca-vf9q2\" (UID: \"c5f8ec4d-adb1-4938-8887-6708463e4b02\") " pod="openshift-image-registry/node-ca-vf9q2" Jul 27 20:34:32.760242 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758502 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b956291a-cecc-490c-8d57-8c1c36575a89-sys-fs\") pod \"aws-ebs-csi-driver-node-j899g\" (UID: \"b956291a-cecc-490c-8d57-8c1c36575a89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j899g" Jul 27 20:34:32.760242 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758524 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw5tw\" (UniqueName: \"kubernetes.io/projected/f0a531c5-371d-4024-9a40-a542d5cfdeb4-kube-api-access-xw5tw\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.760242 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758531 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35f7c023-fb9f-40ea-9f5f-547b26383bb8-cnibin\") pod \"multus-additional-cni-plugins-ljtzv\" (UID: \"35f7c023-fb9f-40ea-9f5f-547b26383bb8\") " pod="openshift-multus/multus-additional-cni-plugins-ljtzv" Jul 27 20:34:32.760242 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758590 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5f8ec4d-adb1-4938-8887-6708463e4b02-host\") pod \"node-ca-vf9q2\" (UID: \"c5f8ec4d-adb1-4938-8887-6708463e4b02\") " pod="openshift-image-registry/node-ca-vf9q2" Jul 27 20:34:32.760242 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758593 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cda0133a-f0f3-4567-bc30-24c8ea984936-env-overrides\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.760242 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758628 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-etc-openvswitch\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.760242 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758700 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/35f7c023-fb9f-40ea-9f5f-547b26383bb8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ljtzv\" (UID: \"35f7c023-fb9f-40ea-9f5f-547b26383bb8\") " pod="openshift-multus/multus-additional-cni-plugins-ljtzv" Jul 27 20:34:32.760242 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758754 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wwq4\" (UniqueName: \"kubernetes.io/projected/e9d570a7-3260-4097-a669-b28a69d3748a-kube-api-access-6wwq4\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.760242 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758787 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpflp\" (UniqueName: \"kubernetes.io/projected/35f7c023-fb9f-40ea-9f5f-547b26383bb8-kube-api-access-fpflp\") pod \"multus-additional-cni-plugins-ljtzv\" (UID: \"35f7c023-fb9f-40ea-9f5f-547b26383bb8\") " pod="openshift-multus/multus-additional-cni-plugins-ljtzv" Jul 27 20:34:32.760242 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758814 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b956291a-cecc-490c-8d57-8c1c36575a89-device-dir\") pod \"aws-ebs-csi-driver-node-j899g\" (UID: \"b956291a-cecc-490c-8d57-8c1c36575a89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j899g" Jul 27 20:34:32.760242 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758840 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b956291a-cecc-490c-8d57-8c1c36575a89-etc-selinux\") pod \"aws-ebs-csi-driver-node-j899g\" (UID: \"b956291a-cecc-490c-8d57-8c1c36575a89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j899g" Jul 27 20:34:32.760242 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758867 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/35f7c023-fb9f-40ea-9f5f-547b26383bb8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ljtzv\" (UID: \"35f7c023-fb9f-40ea-9f5f-547b26383bb8\") " pod="openshift-multus/multus-additional-cni-plugins-ljtzv" Jul 27 20:34:32.760242 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758893 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-var-lib-openvswitch\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.760242 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758918 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f782e3d0-da2b-4b64-ac2e-c50230b51a3a-host-slash\") pod \"iptables-alerter-jhjqt\" (UID: \"f782e3d0-da2b-4b64-ac2e-c50230b51a3a\") " pod="openshift-network-operator/iptables-alerter-jhjqt" Jul 27 20:34:32.760242 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758945 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35f7c023-fb9f-40ea-9f5f-547b26383bb8-cni-binary-copy\") pod \"multus-additional-cni-plugins-ljtzv\" (UID: \"35f7c023-fb9f-40ea-9f5f-547b26383bb8\") " pod="openshift-multus/multus-additional-cni-plugins-ljtzv" Jul 27 20:34:32.760242 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758972 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-run-ovn\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.761105 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.758996 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-host-var-lib-cni-bin\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.761105 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.759021 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e9d570a7-3260-4097-a669-b28a69d3748a-run\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.761105 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.759045 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9d570a7-3260-4097-a669-b28a69d3748a-var-lib-kubelet\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.761105 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.759051 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c5f8ec4d-adb1-4938-8887-6708463e4b02-serviceca\") pod \"node-ca-vf9q2\" (UID: \"c5f8ec4d-adb1-4938-8887-6708463e4b02\") " pod="openshift-image-registry/node-ca-vf9q2" Jul 27 20:34:32.761105 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.759072 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c3561079-4fcf-46be-8d0d-9a20c39be6d9-kubelet-config\") pod \"global-pull-secret-syncer-zrhx5\" (UID: \"c3561079-4fcf-46be-8d0d-9a20c39be6d9\") " pod="kube-system/global-pull-secret-syncer-zrhx5" Jul 27 20:34:32.761105 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.759098 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3561079-4fcf-46be-8d0d-9a20c39be6d9-original-pull-secret\") pod \"global-pull-secret-syncer-zrhx5\" (UID: \"c3561079-4fcf-46be-8d0d-9a20c39be6d9\") " pod="kube-system/global-pull-secret-syncer-zrhx5" Jul 27 20:34:32.761105 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.759108 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-var-lib-openvswitch\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.761105 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.759125 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c122d10-fe63-4972-87c0-4cf309037cc8-tmp-dir\") pod \"node-resolver-2gppc\" (UID: \"8c122d10-fe63-4972-87c0-4cf309037cc8\") " pod="openshift-dns/node-resolver-2gppc" Jul 27 20:34:32.761105 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.759150 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-host-run-netns\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.761105 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.759177 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cda0133a-f0f3-4567-bc30-24c8ea984936-ovnkube-config\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.761105 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.759201 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cda0133a-f0f3-4567-bc30-24c8ea984936-ovnkube-script-lib\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.761105 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.759230 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-system-cni-dir\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.761105 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.759254 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f0a531c5-371d-4024-9a40-a542d5cfdeb4-cni-binary-copy\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.761105 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.759297 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b956291a-cecc-490c-8d57-8c1c36575a89-etc-selinux\") pod \"aws-ebs-csi-driver-node-j899g\" (UID: \"b956291a-cecc-490c-8d57-8c1c36575a89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j899g" Jul 27 20:34:32.761105 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.759340 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b956291a-cecc-490c-8d57-8c1c36575a89-device-dir\") pod \"aws-ebs-csi-driver-node-j899g\" (UID: \"b956291a-cecc-490c-8d57-8c1c36575a89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j899g" Jul 27 20:34:32.761105 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:32.759420 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 20:34:32.761105 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:32.759478 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3561079-4fcf-46be-8d0d-9a20c39be6d9-original-pull-secret podName:c3561079-4fcf-46be-8d0d-9a20c39be6d9 nodeName:}" failed. No retries permitted until 2026-07-27 20:34:33.259457355 +0000 UTC m=+2.995437623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3561079-4fcf-46be-8d0d-9a20c39be6d9-original-pull-secret") pod "global-pull-secret-syncer-zrhx5" (UID: "c3561079-4fcf-46be-8d0d-9a20c39be6d9") : object "kube-system"/"original-pull-secret" not registered Jul 27 20:34:32.762022 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.759484 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-run-ovn\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.762022 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.759519 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c3561079-4fcf-46be-8d0d-9a20c39be6d9-kubelet-config\") pod \"global-pull-secret-syncer-zrhx5\" (UID: \"c3561079-4fcf-46be-8d0d-9a20c39be6d9\") " pod="kube-system/global-pull-secret-syncer-zrhx5" Jul 27 20:34:32.762022 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.759830 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35f7c023-fb9f-40ea-9f5f-547b26383bb8-cni-binary-copy\") pod \"multus-additional-cni-plugins-ljtzv\" (UID: \"35f7c023-fb9f-40ea-9f5f-547b26383bb8\") " pod="openshift-multus/multus-additional-cni-plugins-ljtzv" Jul 27 20:34:32.762022 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.759887 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/35f7c023-fb9f-40ea-9f5f-547b26383bb8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ljtzv\" (UID: \"35f7c023-fb9f-40ea-9f5f-547b26383bb8\") " pod="openshift-multus/multus-additional-cni-plugins-ljtzv" Jul 27 20:34:32.762022 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.759982 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cda0133a-f0f3-4567-bc30-24c8ea984936-ovnkube-config\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.762022 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.760045 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cda0133a-f0f3-4567-bc30-24c8ea984936-host-run-netns\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.762022 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.760186 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c122d10-fe63-4972-87c0-4cf309037cc8-tmp-dir\") pod \"node-resolver-2gppc\" (UID: \"8c122d10-fe63-4972-87c0-4cf309037cc8\") " pod="openshift-dns/node-resolver-2gppc" Jul 27 20:34:32.762022 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.760284 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cda0133a-f0f3-4567-bc30-24c8ea984936-ovnkube-script-lib\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.763035 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.763011 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cda0133a-f0f3-4567-bc30-24c8ea984936-ovn-node-metrics-cert\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.773027 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.772991 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwv5j\" (UniqueName: \"kubernetes.io/projected/b956291a-cecc-490c-8d57-8c1c36575a89-kube-api-access-lwv5j\") pod \"aws-ebs-csi-driver-node-j899g\" (UID: \"b956291a-cecc-490c-8d57-8c1c36575a89\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j899g" Jul 27 20:34:32.774034 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.773876 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9kfx\" (UniqueName: \"kubernetes.io/projected/8c122d10-fe63-4972-87c0-4cf309037cc8-kube-api-access-h9kfx\") pod \"node-resolver-2gppc\" (UID: \"8c122d10-fe63-4972-87c0-4cf309037cc8\") " pod="openshift-dns/node-resolver-2gppc" Jul 27 20:34:32.776313 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.774749 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62m4x\" (UniqueName: \"kubernetes.io/projected/cda0133a-f0f3-4567-bc30-24c8ea984936-kube-api-access-62m4x\") pod \"ovnkube-node-ltxxw\" (UID: \"cda0133a-f0f3-4567-bc30-24c8ea984936\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.776313 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.776206 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpflp\" (UniqueName: \"kubernetes.io/projected/35f7c023-fb9f-40ea-9f5f-547b26383bb8-kube-api-access-fpflp\") pod \"multus-additional-cni-plugins-ljtzv\" (UID: \"35f7c023-fb9f-40ea-9f5f-547b26383bb8\") " pod="openshift-multus/multus-additional-cni-plugins-ljtzv" Jul 27 20:34:32.776495 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.776328 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhx59\" (UniqueName: \"kubernetes.io/projected/c5f8ec4d-adb1-4938-8887-6708463e4b02-kube-api-access-jhx59\") pod \"node-ca-vf9q2\" (UID: \"c5f8ec4d-adb1-4938-8887-6708463e4b02\") " pod="openshift-image-registry/node-ca-vf9q2" Jul 27 20:34:32.852719 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.852634 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-141.ec2.internal" event={"ID":"27125cf5e1c5c4dd4172f89e5d84d2fd","Type":"ContainerStarted","Data":"460e11156864cfca19a35096ae2a91d7e9e09bb55ddd3b17a49509e26ba25ddc"} Jul 27 20:34:32.854534 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.854501 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-141.ec2.internal" event={"ID":"41b9efd1bfa2e292df38ef8f23f5d598","Type":"ContainerStarted","Data":"51f85169de866009428f82bf7101af4edab5a7b7ff271fe23d9b3552bcdd29a3"} Jul 27 20:34:32.855440 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.855410 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 20:34:32.859840 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.859809 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-etc-kubernetes\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.859965 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.859855 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c33a8970-0e52-4e5e-82bf-e8823e1f88d9-konnectivity-ca\") pod \"konnectivity-agent-l6lsr\" (UID: \"c33a8970-0e52-4e5e-82bf-e8823e1f88d9\") " pod="kube-system/konnectivity-agent-l6lsr" Jul 27 20:34:32.859965 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.859886 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw5tw\" (UniqueName: \"kubernetes.io/projected/f0a531c5-371d-4024-9a40-a542d5cfdeb4-kube-api-access-xw5tw\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.859965 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.859912 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wwq4\" (UniqueName: \"kubernetes.io/projected/e9d570a7-3260-4097-a669-b28a69d3748a-kube-api-access-6wwq4\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.860129 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.859966 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-etc-kubernetes\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.860228 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860209 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f782e3d0-da2b-4b64-ac2e-c50230b51a3a-host-slash\") pod \"iptables-alerter-jhjqt\" (UID: \"f782e3d0-da2b-4b64-ac2e-c50230b51a3a\") " pod="openshift-network-operator/iptables-alerter-jhjqt" Jul 27 20:34:32.860299 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860246 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-host-var-lib-cni-bin\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.860299 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860276 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e9d570a7-3260-4097-a669-b28a69d3748a-run\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.860403 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860302 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9d570a7-3260-4097-a669-b28a69d3748a-var-lib-kubelet\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.860403 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860344 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-system-cni-dir\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.860403 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860366 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-host-var-lib-cni-bin\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.860557 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860370 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f0a531c5-371d-4024-9a40-a542d5cfdeb4-cni-binary-copy\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.860557 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860437 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-cnibin\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.860557 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860463 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e9d570a7-3260-4097-a669-b28a69d3748a-etc-sysconfig\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.860557 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860485 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c33a8970-0e52-4e5e-82bf-e8823e1f88d9-konnectivity-ca\") pod \"konnectivity-agent-l6lsr\" (UID: \"c33a8970-0e52-4e5e-82bf-e8823e1f88d9\") " pod="kube-system/konnectivity-agent-l6lsr" Jul 27 20:34:32.860557 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860489 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e9d570a7-3260-4097-a669-b28a69d3748a-etc-systemd\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.860557 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860548 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9d570a7-3260-4097-a669-b28a69d3748a-sys\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.860888 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860558 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e9d570a7-3260-4097-a669-b28a69d3748a-etc-systemd\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.860888 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860580 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6dcq\" (UniqueName: \"kubernetes.io/projected/58bd29b9-0422-4c07-90f2-04d39cd07db5-kube-api-access-m6dcq\") pod \"network-check-target-j42rs\" (UID: \"58bd29b9-0422-4c07-90f2-04d39cd07db5\") " pod="openshift-network-diagnostics/network-check-target-j42rs" Jul 27 20:34:32.860888 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860603 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-host-run-netns\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.860888 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860631 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05ef8a1e-7bd9-4436-a007-618d151dfd5a-metrics-certs\") pod \"network-metrics-daemon-tkd2d\" (UID: \"05ef8a1e-7bd9-4436-a007-618d151dfd5a\") " pod="openshift-multus/network-metrics-daemon-tkd2d" Jul 27 20:34:32.860888 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860651 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e9d570a7-3260-4097-a669-b28a69d3748a-etc-modprobe-d\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.860888 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860650 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-cnibin\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.860888 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860667 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f782e3d0-da2b-4b64-ac2e-c50230b51a3a-iptables-alerter-script\") pod \"iptables-alerter-jhjqt\" (UID: \"f782e3d0-da2b-4b64-ac2e-c50230b51a3a\") " pod="openshift-network-operator/iptables-alerter-jhjqt" Jul 27 20:34:32.860888 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860698 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9d570a7-3260-4097-a669-b28a69d3748a-host\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.860888 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860705 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e9d570a7-3260-4097-a669-b28a69d3748a-etc-sysconfig\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.860888 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860462 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f782e3d0-da2b-4b64-ac2e-c50230b51a3a-host-slash\") pod \"iptables-alerter-jhjqt\" (UID: \"f782e3d0-da2b-4b64-ac2e-c50230b51a3a\") " pod="openshift-network-operator/iptables-alerter-jhjqt" Jul 27 20:34:32.860888 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860717 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-hostroot\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.860888 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860749 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m8td\" (UniqueName: \"kubernetes.io/projected/05ef8a1e-7bd9-4436-a007-618d151dfd5a-kube-api-access-5m8td\") pod \"network-metrics-daemon-tkd2d\" (UID: \"05ef8a1e-7bd9-4436-a007-618d151dfd5a\") " pod="openshift-multus/network-metrics-daemon-tkd2d" Jul 27 20:34:32.860888 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860773 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e9d570a7-3260-4097-a669-b28a69d3748a-etc-sysctl-d\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.860888 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860775 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9d570a7-3260-4097-a669-b28a69d3748a-var-lib-kubelet\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.860888 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860799 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-host-var-lib-cni-multus\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.860888 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860822 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e9d570a7-3260-4097-a669-b28a69d3748a-etc-sysctl-conf\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.860888 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860838 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9d570a7-3260-4097-a669-b28a69d3748a-sys\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.860888 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860846 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9d570a7-3260-4097-a669-b28a69d3748a-tmp\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.861780 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860856 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f0a531c5-371d-4024-9a40-a542d5cfdeb4-cni-binary-copy\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.861780 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860873 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f0a531c5-371d-4024-9a40-a542d5cfdeb4-multus-daemon-config\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.861780 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860880 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-hostroot\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.861780 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860905 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-host-run-netns\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.861780 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860922 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-host-run-multus-certs\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.861780 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860931 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e9d570a7-3260-4097-a669-b28a69d3748a-run\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.861780 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860925 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-system-cni-dir\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.861780 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860949 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48mh6\" (UniqueName: \"kubernetes.io/projected/f782e3d0-da2b-4b64-ac2e-c50230b51a3a-kube-api-access-48mh6\") pod \"iptables-alerter-jhjqt\" (UID: \"f782e3d0-da2b-4b64-ac2e-c50230b51a3a\") " pod="openshift-network-operator/iptables-alerter-jhjqt" Jul 27 20:34:32.861780 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.860962 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-host-var-lib-cni-multus\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.861780 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.861005 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9d570a7-3260-4097-a669-b28a69d3748a-host\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.861780 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.861012 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-host-run-multus-certs\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.861780 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.861084 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e9d570a7-3260-4097-a669-b28a69d3748a-etc-modprobe-d\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.861780 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:32.860964 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 20:34:32.861780 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:32.861209 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05ef8a1e-7bd9-4436-a007-618d151dfd5a-metrics-certs podName:05ef8a1e-7bd9-4436-a007-618d151dfd5a nodeName:}" failed. No retries permitted until 2026-07-27 20:34:33.361189866 +0000 UTC m=+3.097170137 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05ef8a1e-7bd9-4436-a007-618d151dfd5a-metrics-certs") pod "network-metrics-daemon-tkd2d" (UID: "05ef8a1e-7bd9-4436-a007-618d151dfd5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 20:34:32.861780 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.861384 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e9d570a7-3260-4097-a669-b28a69d3748a-etc-sysctl-d\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.861780 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.861401 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f0a531c5-371d-4024-9a40-a542d5cfdeb4-multus-daemon-config\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.861780 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.861423 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-multus-socket-dir-parent\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.861780 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.861449 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-multus-conf-dir\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.862786 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.861478 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-host-run-k8s-cni-cncf-io\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.862786 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.861501 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e9d570a7-3260-4097-a669-b28a69d3748a-etc-sysctl-conf\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.862786 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.861503 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9d570a7-3260-4097-a669-b28a69d3748a-lib-modules\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.862786 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.861533 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-multus-conf-dir\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.862786 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.861541 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-multus-cni-dir\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.862786 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.861571 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c33a8970-0e52-4e5e-82bf-e8823e1f88d9-agent-certs\") pod \"konnectivity-agent-l6lsr\" (UID: \"c33a8970-0e52-4e5e-82bf-e8823e1f88d9\") " pod="kube-system/konnectivity-agent-l6lsr" Jul 27 20:34:32.862786 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.861598 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-multus-cni-dir\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.862786 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.861604 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9d570a7-3260-4097-a669-b28a69d3748a-lib-modules\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.862786 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.861628 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9d570a7-3260-4097-a669-b28a69d3748a-etc-kubernetes\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.862786 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.861653 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-host-run-k8s-cni-cncf-io\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.862786 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.861659 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-os-release\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.862786 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.861672 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-multus-socket-dir-parent\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.862786 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.861679 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-host-var-lib-kubelet\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.862786 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.861706 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e9d570a7-3260-4097-a669-b28a69d3748a-etc-tuned\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.862786 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.861710 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-host-var-lib-kubelet\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.862786 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.861771 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f782e3d0-da2b-4b64-ac2e-c50230b51a3a-iptables-alerter-script\") pod \"iptables-alerter-jhjqt\" (UID: \"f782e3d0-da2b-4b64-ac2e-c50230b51a3a\") " pod="openshift-network-operator/iptables-alerter-jhjqt" Jul 27 20:34:32.862786 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.861791 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f0a531c5-371d-4024-9a40-a542d5cfdeb4-os-release\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.862786 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.861832 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9d570a7-3260-4097-a669-b28a69d3748a-etc-kubernetes\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.865769 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.865745 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e9d570a7-3260-4097-a669-b28a69d3748a-etc-tuned\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.865880 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.865777 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c33a8970-0e52-4e5e-82bf-e8823e1f88d9-agent-certs\") pod \"konnectivity-agent-l6lsr\" (UID: \"c33a8970-0e52-4e5e-82bf-e8823e1f88d9\") " pod="kube-system/konnectivity-agent-l6lsr" Jul 27 20:34:32.867728 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.867706 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9d570a7-3260-4097-a669-b28a69d3748a-tmp\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.869800 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:32.869779 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 20:34:32.869885 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:32.869808 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 20:34:32.869885 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:32.869823 2570 projected.go:196] Error preparing data for projected volume kube-api-access-m6dcq for pod openshift-network-diagnostics/network-check-target-j42rs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 20:34:32.870006 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:32.869886 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58bd29b9-0422-4c07-90f2-04d39cd07db5-kube-api-access-m6dcq podName:58bd29b9-0422-4c07-90f2-04d39cd07db5 nodeName:}" failed. No retries permitted until 2026-07-27 20:34:33.369866605 +0000 UTC m=+3.105846896 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-m6dcq" (UniqueName: "kubernetes.io/projected/58bd29b9-0422-4c07-90f2-04d39cd07db5-kube-api-access-m6dcq") pod "network-check-target-j42rs" (UID: "58bd29b9-0422-4c07-90f2-04d39cd07db5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 20:34:32.873799 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.873762 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw5tw\" (UniqueName: \"kubernetes.io/projected/f0a531c5-371d-4024-9a40-a542d5cfdeb4-kube-api-access-xw5tw\") pod \"multus-22mpg\" (UID: \"f0a531c5-371d-4024-9a40-a542d5cfdeb4\") " pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.874861 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.874835 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wwq4\" (UniqueName: \"kubernetes.io/projected/e9d570a7-3260-4097-a669-b28a69d3748a-kube-api-access-6wwq4\") pod \"tuned-5dp7l\" (UID: \"e9d570a7-3260-4097-a669-b28a69d3748a\") " pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:32.877211 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.877152 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48mh6\" (UniqueName: \"kubernetes.io/projected/f782e3d0-da2b-4b64-ac2e-c50230b51a3a-kube-api-access-48mh6\") pod \"iptables-alerter-jhjqt\" (UID: \"f782e3d0-da2b-4b64-ac2e-c50230b51a3a\") " pod="openshift-network-operator/iptables-alerter-jhjqt" Jul 27 20:34:32.878877 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.878858 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m8td\" (UniqueName: \"kubernetes.io/projected/05ef8a1e-7bd9-4436-a007-618d151dfd5a-kube-api-access-5m8td\") pod \"network-metrics-daemon-tkd2d\" (UID: \"05ef8a1e-7bd9-4436-a007-618d151dfd5a\") " pod="openshift-multus/network-metrics-daemon-tkd2d" Jul 27 20:34:32.950971 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.950925 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ljtzv" Jul 27 20:34:32.960350 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.960144 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vf9q2" Jul 27 20:34:32.977110 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:32.976184 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35f7c023_fb9f_40ea_9f5f_547b26383bb8.slice/crio-2f0b22d39b65a9966d38b22c87c7699f1181ea1da7c107ca28bd71e4ff287ca3 WatchSource:0}: Error finding container 2f0b22d39b65a9966d38b22c87c7699f1181ea1da7c107ca28bd71e4ff287ca3: Status 404 returned error can't find the container with id 2f0b22d39b65a9966d38b22c87c7699f1181ea1da7c107ca28bd71e4ff287ca3 Jul 27 20:34:32.977110 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.976462 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:32.977110 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.976670 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j899g" Jul 27 20:34:32.987605 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.987583 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2gppc" Jul 27 20:34:32.996823 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.996795 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-22mpg" Jul 27 20:34:32.999980 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:32.999753 2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 20:34:33.006184 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:33.006140 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb956291a_cecc_490c_8d57_8c1c36575a89.slice/crio-cb407757a72fcc7208439fc739135dad027ffcf00a7ec36a441affd828ab5f32 WatchSource:0}: Error finding container cb407757a72fcc7208439fc739135dad027ffcf00a7ec36a441affd828ab5f32: Status 404 returned error can't find the container with id cb407757a72fcc7208439fc739135dad027ffcf00a7ec36a441affd828ab5f32 Jul 27 20:34:33.006996 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:33.006965 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jhjqt" Jul 27 20:34:33.016215 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:33.015849 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l6lsr" Jul 27 20:34:33.023922 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:33.023896 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" Jul 27 20:34:33.026353 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:33.026323 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc33a8970_0e52_4e5e_82bf_e8823e1f88d9.slice/crio-50a604fb2aad3b672d363c536715ffb2f01dbe28c5a21f621fcec4235e5517d7 WatchSource:0}: Error finding container 50a604fb2aad3b672d363c536715ffb2f01dbe28c5a21f621fcec4235e5517d7: Status 404 returned error can't find the container with id 50a604fb2aad3b672d363c536715ffb2f01dbe28c5a21f621fcec4235e5517d7 Jul 27 20:34:33.036461 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:34:33.036424 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9d570a7_3260_4097_a669_b28a69d3748a.slice/crio-c3b273d7c12f7eed40ef4cb0758b288ebddd9dac91c5938336ae7178a90d2789 WatchSource:0}: Error finding container c3b273d7c12f7eed40ef4cb0758b288ebddd9dac91c5938336ae7178a90d2789: Status 404 returned error can't find the container with id c3b273d7c12f7eed40ef4cb0758b288ebddd9dac91c5938336ae7178a90d2789 Jul 27 20:34:33.265210 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:33.265109 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3561079-4fcf-46be-8d0d-9a20c39be6d9-original-pull-secret\") pod \"global-pull-secret-syncer-zrhx5\" (UID: \"c3561079-4fcf-46be-8d0d-9a20c39be6d9\") " pod="kube-system/global-pull-secret-syncer-zrhx5" Jul 27 20:34:33.265362 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:33.265287 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 20:34:33.265418 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:33.265371 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3561079-4fcf-46be-8d0d-9a20c39be6d9-original-pull-secret podName:c3561079-4fcf-46be-8d0d-9a20c39be6d9 nodeName:}" failed. No retries permitted until 2026-07-27 20:34:34.265347946 +0000 UTC m=+4.001328258 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3561079-4fcf-46be-8d0d-9a20c39be6d9-original-pull-secret") pod "global-pull-secret-syncer-zrhx5" (UID: "c3561079-4fcf-46be-8d0d-9a20c39be6d9") : object "kube-system"/"original-pull-secret" not registered Jul 27 20:34:33.365836 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:33.365791 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05ef8a1e-7bd9-4436-a007-618d151dfd5a-metrics-certs\") pod \"network-metrics-daemon-tkd2d\" (UID: \"05ef8a1e-7bd9-4436-a007-618d151dfd5a\") " pod="openshift-multus/network-metrics-daemon-tkd2d" Jul 27 20:34:33.366017 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:33.365943 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 20:34:33.366017 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:33.366010 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05ef8a1e-7bd9-4436-a007-618d151dfd5a-metrics-certs podName:05ef8a1e-7bd9-4436-a007-618d151dfd5a nodeName:}" failed. No retries permitted until 2026-07-27 20:34:34.3659938 +0000 UTC m=+4.101974066 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05ef8a1e-7bd9-4436-a007-618d151dfd5a-metrics-certs") pod "network-metrics-daemon-tkd2d" (UID: "05ef8a1e-7bd9-4436-a007-618d151dfd5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 20:34:33.466844 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:33.466807 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6dcq\" (UniqueName: \"kubernetes.io/projected/58bd29b9-0422-4c07-90f2-04d39cd07db5-kube-api-access-m6dcq\") pod \"network-check-target-j42rs\" (UID: \"58bd29b9-0422-4c07-90f2-04d39cd07db5\") " pod="openshift-network-diagnostics/network-check-target-j42rs" Jul 27 20:34:33.467029 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:33.467006 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 20:34:33.467079 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:33.467036 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 20:34:33.467079 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:33.467047 2570 projected.go:196] Error preparing data for projected volume kube-api-access-m6dcq for pod openshift-network-diagnostics/network-check-target-j42rs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 20:34:33.467158 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:33.467105 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58bd29b9-0422-4c07-90f2-04d39cd07db5-kube-api-access-m6dcq podName:58bd29b9-0422-4c07-90f2-04d39cd07db5 nodeName:}" failed. No retries permitted until 2026-07-27 20:34:34.467088533 +0000 UTC m=+4.203068801 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-m6dcq" (UniqueName: "kubernetes.io/projected/58bd29b9-0422-4c07-90f2-04d39cd07db5-kube-api-access-m6dcq") pod "network-check-target-j42rs" (UID: "58bd29b9-0422-4c07-90f2-04d39cd07db5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 20:34:33.703797 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:33.703594 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 20:29:33 +0000 UTC" deadline="2028-02-13 17:19:22.463745779 +0000 UTC" Jul 27 20:34:33.703797 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:33.703681 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13580h44m48.760069186s" Jul 27 20:34:33.847722 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:33.847682 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j42rs" Jul 27 20:34:33.848094 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:33.847832 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j42rs" podUID="58bd29b9-0422-4c07-90f2-04d39cd07db5" Jul 27 20:34:33.859649 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:33.859609 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" event={"ID":"e9d570a7-3260-4097-a669-b28a69d3748a","Type":"ContainerStarted","Data":"c3b273d7c12f7eed40ef4cb0758b288ebddd9dac91c5938336ae7178a90d2789"} Jul 27 20:34:33.862231 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:33.862200 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-22mpg" event={"ID":"f0a531c5-371d-4024-9a40-a542d5cfdeb4","Type":"ContainerStarted","Data":"a262dfe2eacf3a960205f45c2c3567e44920b56343592f905d2e3541ce2b9978"} Jul 27 20:34:33.864943 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:33.864910 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" event={"ID":"cda0133a-f0f3-4567-bc30-24c8ea984936","Type":"ContainerStarted","Data":"4a8e8ca2f8003523969c2150968392967b36874fe8d2ddf900d39fc3fdd0dce2"} Jul 27 20:34:33.868399 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:33.868368 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vf9q2" event={"ID":"c5f8ec4d-adb1-4938-8887-6708463e4b02","Type":"ContainerStarted","Data":"bfb4764fc6582bdbe36516c0e893ea551adfe4592bc91b06a6cfd4cca9755748"} Jul 27 20:34:33.870577 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:33.870547 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ljtzv" event={"ID":"35f7c023-fb9f-40ea-9f5f-547b26383bb8","Type":"ContainerStarted","Data":"2f0b22d39b65a9966d38b22c87c7699f1181ea1da7c107ca28bd71e4ff287ca3"} Jul 27 20:34:33.872121 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:33.872091 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l6lsr" event={"ID":"c33a8970-0e52-4e5e-82bf-e8823e1f88d9","Type":"ContainerStarted","Data":"50a604fb2aad3b672d363c536715ffb2f01dbe28c5a21f621fcec4235e5517d7"} Jul 27 20:34:33.874422 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:33.874395 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jhjqt" event={"ID":"f782e3d0-da2b-4b64-ac2e-c50230b51a3a","Type":"ContainerStarted","Data":"326ad4cac58618f773cc913fd035618e195c431b6b1515c5de45644db7185030"} Jul 27 20:34:33.878631 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:33.878603 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2gppc" event={"ID":"8c122d10-fe63-4972-87c0-4cf309037cc8","Type":"ContainerStarted","Data":"73a2f8ee9709bb5a5fa532f221eb3c5e36eccc7db85b7ac13e47c91d7c47ae9c"} Jul 27 20:34:33.880939 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:33.880911 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j899g" event={"ID":"b956291a-cecc-490c-8d57-8c1c36575a89","Type":"ContainerStarted","Data":"cb407757a72fcc7208439fc739135dad027ffcf00a7ec36a441affd828ab5f32"} Jul 27 20:34:34.272794 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:34.272747 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3561079-4fcf-46be-8d0d-9a20c39be6d9-original-pull-secret\") pod \"global-pull-secret-syncer-zrhx5\" (UID: \"c3561079-4fcf-46be-8d0d-9a20c39be6d9\") " pod="kube-system/global-pull-secret-syncer-zrhx5" Jul 27 20:34:34.272994 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:34.272973 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 20:34:34.273120 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:34.273053 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3561079-4fcf-46be-8d0d-9a20c39be6d9-original-pull-secret podName:c3561079-4fcf-46be-8d0d-9a20c39be6d9 nodeName:}" failed. No retries permitted until 2026-07-27 20:34:36.273032866 +0000 UTC m=+6.009013134 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3561079-4fcf-46be-8d0d-9a20c39be6d9-original-pull-secret") pod "global-pull-secret-syncer-zrhx5" (UID: "c3561079-4fcf-46be-8d0d-9a20c39be6d9") : object "kube-system"/"original-pull-secret" not registered Jul 27 20:34:34.374205 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:34.374164 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05ef8a1e-7bd9-4436-a007-618d151dfd5a-metrics-certs\") pod \"network-metrics-daemon-tkd2d\" (UID: \"05ef8a1e-7bd9-4436-a007-618d151dfd5a\") " pod="openshift-multus/network-metrics-daemon-tkd2d" Jul 27 20:34:34.374478 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:34.374346 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 20:34:34.374478 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:34.374433 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05ef8a1e-7bd9-4436-a007-618d151dfd5a-metrics-certs podName:05ef8a1e-7bd9-4436-a007-618d151dfd5a nodeName:}" failed. No retries permitted until 2026-07-27 20:34:36.374409998 +0000 UTC m=+6.110390282 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05ef8a1e-7bd9-4436-a007-618d151dfd5a-metrics-certs") pod "network-metrics-daemon-tkd2d" (UID: "05ef8a1e-7bd9-4436-a007-618d151dfd5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 20:34:34.475125 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:34.475082 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6dcq\" (UniqueName: \"kubernetes.io/projected/58bd29b9-0422-4c07-90f2-04d39cd07db5-kube-api-access-m6dcq\") pod \"network-check-target-j42rs\" (UID: \"58bd29b9-0422-4c07-90f2-04d39cd07db5\") " pod="openshift-network-diagnostics/network-check-target-j42rs" Jul 27 20:34:34.475319 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:34.475299 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 20:34:34.475395 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:34.475323 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 20:34:34.475395 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:34.475337 2570 projected.go:196] Error preparing data for projected volume kube-api-access-m6dcq for pod openshift-network-diagnostics/network-check-target-j42rs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 20:34:34.475513 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:34.475412 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58bd29b9-0422-4c07-90f2-04d39cd07db5-kube-api-access-m6dcq podName:58bd29b9-0422-4c07-90f2-04d39cd07db5 nodeName:}" failed. No retries permitted until 2026-07-27 20:34:36.475392376 +0000 UTC m=+6.211372646 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-m6dcq" (UniqueName: "kubernetes.io/projected/58bd29b9-0422-4c07-90f2-04d39cd07db5-kube-api-access-m6dcq") pod "network-check-target-j42rs" (UID: "58bd29b9-0422-4c07-90f2-04d39cd07db5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 20:34:34.704230 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:34.704092 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 20:29:33 +0000 UTC" deadline="2028-03-13 18:38:53.201289896 +0000 UTC" Jul 27 20:34:34.704230 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:34.704140 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14278h4m18.497155728s" Jul 27 20:34:34.850993 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:34.850063 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkd2d" Jul 27 20:34:34.850993 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:34.850470 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkd2d" podUID="05ef8a1e-7bd9-4436-a007-618d151dfd5a" Jul 27 20:34:34.850993 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:34.850789 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrhx5" Jul 27 20:34:34.850993 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:34.850883 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrhx5" podUID="c3561079-4fcf-46be-8d0d-9a20c39be6d9" Jul 27 20:34:35.847908 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:35.847266 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j42rs" Jul 27 20:34:35.847908 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:35.847433 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j42rs" podUID="58bd29b9-0422-4c07-90f2-04d39cd07db5" Jul 27 20:34:36.294906 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:36.294179 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3561079-4fcf-46be-8d0d-9a20c39be6d9-original-pull-secret\") pod \"global-pull-secret-syncer-zrhx5\" (UID: \"c3561079-4fcf-46be-8d0d-9a20c39be6d9\") " pod="kube-system/global-pull-secret-syncer-zrhx5" Jul 27 20:34:36.294906 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:36.294352 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 20:34:36.294906 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:36.294420 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3561079-4fcf-46be-8d0d-9a20c39be6d9-original-pull-secret podName:c3561079-4fcf-46be-8d0d-9a20c39be6d9 nodeName:}" failed. No retries permitted until 2026-07-27 20:34:40.2944002 +0000 UTC m=+10.030380485 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3561079-4fcf-46be-8d0d-9a20c39be6d9-original-pull-secret") pod "global-pull-secret-syncer-zrhx5" (UID: "c3561079-4fcf-46be-8d0d-9a20c39be6d9") : object "kube-system"/"original-pull-secret" not registered Jul 27 20:34:36.395520 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:36.394877 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05ef8a1e-7bd9-4436-a007-618d151dfd5a-metrics-certs\") pod \"network-metrics-daemon-tkd2d\" (UID: \"05ef8a1e-7bd9-4436-a007-618d151dfd5a\") " pod="openshift-multus/network-metrics-daemon-tkd2d" Jul 27 20:34:36.395520 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:36.395056 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 20:34:36.395520 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:36.395131 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05ef8a1e-7bd9-4436-a007-618d151dfd5a-metrics-certs podName:05ef8a1e-7bd9-4436-a007-618d151dfd5a nodeName:}" failed. No retries permitted until 2026-07-27 20:34:40.39511046 +0000 UTC m=+10.131090731 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05ef8a1e-7bd9-4436-a007-618d151dfd5a-metrics-certs") pod "network-metrics-daemon-tkd2d" (UID: "05ef8a1e-7bd9-4436-a007-618d151dfd5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 20:34:36.496018 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:36.495314 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6dcq\" (UniqueName: \"kubernetes.io/projected/58bd29b9-0422-4c07-90f2-04d39cd07db5-kube-api-access-m6dcq\") pod \"network-check-target-j42rs\" (UID: \"58bd29b9-0422-4c07-90f2-04d39cd07db5\") " pod="openshift-network-diagnostics/network-check-target-j42rs" Jul 27 20:34:36.496018 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:36.495530 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 20:34:36.496018 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:36.495556 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 20:34:36.496018 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:36.495570 2570 projected.go:196] Error preparing data for projected volume kube-api-access-m6dcq for pod openshift-network-diagnostics/network-check-target-j42rs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 20:34:36.496018 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:36.495646 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58bd29b9-0422-4c07-90f2-04d39cd07db5-kube-api-access-m6dcq podName:58bd29b9-0422-4c07-90f2-04d39cd07db5 nodeName:}" failed. No retries permitted until 2026-07-27 20:34:40.49562589 +0000 UTC m=+10.231606159 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-m6dcq" (UniqueName: "kubernetes.io/projected/58bd29b9-0422-4c07-90f2-04d39cd07db5-kube-api-access-m6dcq") pod "network-check-target-j42rs" (UID: "58bd29b9-0422-4c07-90f2-04d39cd07db5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 20:34:36.848926 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:36.848878 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrhx5" Jul 27 20:34:36.849435 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:36.849052 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrhx5" podUID="c3561079-4fcf-46be-8d0d-9a20c39be6d9" Jul 27 20:34:36.849580 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:36.849556 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkd2d" Jul 27 20:34:36.849743 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:36.849718 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkd2d" podUID="05ef8a1e-7bd9-4436-a007-618d151dfd5a" Jul 27 20:34:37.847858 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:37.847265 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j42rs" Jul 27 20:34:37.847858 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:37.847418 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j42rs" podUID="58bd29b9-0422-4c07-90f2-04d39cd07db5" Jul 27 20:34:38.851384 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:38.851293 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrhx5" Jul 27 20:34:38.851871 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:38.851434 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrhx5" podUID="c3561079-4fcf-46be-8d0d-9a20c39be6d9" Jul 27 20:34:38.851871 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:38.851819 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkd2d" Jul 27 20:34:38.851981 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:38.851943 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkd2d" podUID="05ef8a1e-7bd9-4436-a007-618d151dfd5a" Jul 27 20:34:39.847231 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:39.847188 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j42rs" Jul 27 20:34:39.847443 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:39.847335 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j42rs" podUID="58bd29b9-0422-4c07-90f2-04d39cd07db5" Jul 27 20:34:40.327099 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:40.326951 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3561079-4fcf-46be-8d0d-9a20c39be6d9-original-pull-secret\") pod \"global-pull-secret-syncer-zrhx5\" (UID: \"c3561079-4fcf-46be-8d0d-9a20c39be6d9\") " pod="kube-system/global-pull-secret-syncer-zrhx5" Jul 27 20:34:40.327595 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:40.327133 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 20:34:40.327595 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:40.327216 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3561079-4fcf-46be-8d0d-9a20c39be6d9-original-pull-secret podName:c3561079-4fcf-46be-8d0d-9a20c39be6d9 nodeName:}" failed. No retries permitted until 2026-07-27 20:34:48.327192718 +0000 UTC m=+18.063172990 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3561079-4fcf-46be-8d0d-9a20c39be6d9-original-pull-secret") pod "global-pull-secret-syncer-zrhx5" (UID: "c3561079-4fcf-46be-8d0d-9a20c39be6d9") : object "kube-system"/"original-pull-secret" not registered Jul 27 20:34:40.428034 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:40.427977 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05ef8a1e-7bd9-4436-a007-618d151dfd5a-metrics-certs\") pod \"network-metrics-daemon-tkd2d\" (UID: \"05ef8a1e-7bd9-4436-a007-618d151dfd5a\") " pod="openshift-multus/network-metrics-daemon-tkd2d" Jul 27 20:34:40.428214 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:40.428140 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 20:34:40.428214 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:40.428207 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05ef8a1e-7bd9-4436-a007-618d151dfd5a-metrics-certs podName:05ef8a1e-7bd9-4436-a007-618d151dfd5a nodeName:}" failed. No retries permitted until 2026-07-27 20:34:48.428186964 +0000 UTC m=+18.164167245 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05ef8a1e-7bd9-4436-a007-618d151dfd5a-metrics-certs") pod "network-metrics-daemon-tkd2d" (UID: "05ef8a1e-7bd9-4436-a007-618d151dfd5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 20:34:40.530143 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:40.529413 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6dcq\" (UniqueName: \"kubernetes.io/projected/58bd29b9-0422-4c07-90f2-04d39cd07db5-kube-api-access-m6dcq\") pod \"network-check-target-j42rs\" (UID: \"58bd29b9-0422-4c07-90f2-04d39cd07db5\") " pod="openshift-network-diagnostics/network-check-target-j42rs" Jul 27 20:34:40.530143 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:40.529614 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 20:34:40.530143 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:40.529656 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 20:34:40.530143 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:40.529678 2570 projected.go:196] Error preparing data for projected volume kube-api-access-m6dcq for pod openshift-network-diagnostics/network-check-target-j42rs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 20:34:40.530143 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:40.529743 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58bd29b9-0422-4c07-90f2-04d39cd07db5-kube-api-access-m6dcq podName:58bd29b9-0422-4c07-90f2-04d39cd07db5 nodeName:}" failed. No retries permitted until 2026-07-27 20:34:48.529721716 +0000 UTC m=+18.265701988 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-m6dcq" (UniqueName: "kubernetes.io/projected/58bd29b9-0422-4c07-90f2-04d39cd07db5-kube-api-access-m6dcq") pod "network-check-target-j42rs" (UID: "58bd29b9-0422-4c07-90f2-04d39cd07db5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 20:34:40.849209 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:40.848722 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrhx5" Jul 27 20:34:40.849209 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:40.848861 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrhx5" podUID="c3561079-4fcf-46be-8d0d-9a20c39be6d9" Jul 27 20:34:40.849209 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:40.848920 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkd2d" Jul 27 20:34:40.849209 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:40.849074 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkd2d" podUID="05ef8a1e-7bd9-4436-a007-618d151dfd5a" Jul 27 20:34:41.847550 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:41.847509 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j42rs" Jul 27 20:34:41.848083 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:41.847669 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j42rs" podUID="58bd29b9-0422-4c07-90f2-04d39cd07db5" Jul 27 20:34:42.848726 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:42.848699 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkd2d" Jul 27 20:34:42.849199 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:42.848804 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkd2d" podUID="05ef8a1e-7bd9-4436-a007-618d151dfd5a" Jul 27 20:34:42.849784 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:42.849747 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrhx5" Jul 27 20:34:42.849892 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:42.849870 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrhx5" podUID="c3561079-4fcf-46be-8d0d-9a20c39be6d9" Jul 27 20:34:43.847118 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:43.847082 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j42rs" Jul 27 20:34:43.847295 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:43.847199 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j42rs" podUID="58bd29b9-0422-4c07-90f2-04d39cd07db5" Jul 27 20:34:44.847905 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:44.847866 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrhx5" Jul 27 20:34:44.847905 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:44.847898 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkd2d" Jul 27 20:34:44.848385 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:44.848012 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrhx5" podUID="c3561079-4fcf-46be-8d0d-9a20c39be6d9" Jul 27 20:34:44.848385 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:44.848114 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkd2d" podUID="05ef8a1e-7bd9-4436-a007-618d151dfd5a" Jul 27 20:34:45.846853 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:45.846804 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j42rs" Jul 27 20:34:45.847035 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:45.846933 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j42rs" podUID="58bd29b9-0422-4c07-90f2-04d39cd07db5" Jul 27 20:34:46.847511 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:46.847475 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrhx5" Jul 27 20:34:46.847994 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:46.847490 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkd2d" Jul 27 20:34:46.847994 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:46.847614 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrhx5" podUID="c3561079-4fcf-46be-8d0d-9a20c39be6d9" Jul 27 20:34:46.847994 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:46.847719 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkd2d" podUID="05ef8a1e-7bd9-4436-a007-618d151dfd5a" Jul 27 20:34:47.847309 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:47.847266 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j42rs" Jul 27 20:34:47.847494 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:47.847418 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j42rs" podUID="58bd29b9-0422-4c07-90f2-04d39cd07db5" Jul 27 20:34:48.394669 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:48.394633 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3561079-4fcf-46be-8d0d-9a20c39be6d9-original-pull-secret\") pod \"global-pull-secret-syncer-zrhx5\" (UID: \"c3561079-4fcf-46be-8d0d-9a20c39be6d9\") " pod="kube-system/global-pull-secret-syncer-zrhx5" Jul 27 20:34:48.395123 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:48.394811 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 20:34:48.395123 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:48.394881 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3561079-4fcf-46be-8d0d-9a20c39be6d9-original-pull-secret podName:c3561079-4fcf-46be-8d0d-9a20c39be6d9 nodeName:}" failed. No retries permitted until 2026-07-27 20:35:04.394863202 +0000 UTC m=+34.130843470 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3561079-4fcf-46be-8d0d-9a20c39be6d9-original-pull-secret") pod "global-pull-secret-syncer-zrhx5" (UID: "c3561079-4fcf-46be-8d0d-9a20c39be6d9") : object "kube-system"/"original-pull-secret" not registered Jul 27 20:34:48.496085 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:48.496040 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05ef8a1e-7bd9-4436-a007-618d151dfd5a-metrics-certs\") pod \"network-metrics-daemon-tkd2d\" (UID: \"05ef8a1e-7bd9-4436-a007-618d151dfd5a\") " pod="openshift-multus/network-metrics-daemon-tkd2d" Jul 27 20:34:48.496270 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:48.496201 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 20:34:48.496322 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:48.496285 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05ef8a1e-7bd9-4436-a007-618d151dfd5a-metrics-certs podName:05ef8a1e-7bd9-4436-a007-618d151dfd5a nodeName:}" failed. No retries permitted until 2026-07-27 20:35:04.496262787 +0000 UTC m=+34.232243056 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05ef8a1e-7bd9-4436-a007-618d151dfd5a-metrics-certs") pod "network-metrics-daemon-tkd2d" (UID: "05ef8a1e-7bd9-4436-a007-618d151dfd5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 20:34:48.597159 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:48.597117 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6dcq\" (UniqueName: \"kubernetes.io/projected/58bd29b9-0422-4c07-90f2-04d39cd07db5-kube-api-access-m6dcq\") pod \"network-check-target-j42rs\" (UID: \"58bd29b9-0422-4c07-90f2-04d39cd07db5\") " pod="openshift-network-diagnostics/network-check-target-j42rs" Jul 27 20:34:48.597350 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:48.597312 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 20:34:48.597350 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:48.597341 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 20:34:48.597463 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:48.597355 2570 projected.go:196] Error preparing data for projected volume kube-api-access-m6dcq for pod openshift-network-diagnostics/network-check-target-j42rs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 20:34:48.597463 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:48.597423 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58bd29b9-0422-4c07-90f2-04d39cd07db5-kube-api-access-m6dcq podName:58bd29b9-0422-4c07-90f2-04d39cd07db5 nodeName:}" failed. No retries permitted until 2026-07-27 20:35:04.597401866 +0000 UTC m=+34.333382453 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-m6dcq" (UniqueName: "kubernetes.io/projected/58bd29b9-0422-4c07-90f2-04d39cd07db5-kube-api-access-m6dcq") pod "network-check-target-j42rs" (UID: "58bd29b9-0422-4c07-90f2-04d39cd07db5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 20:34:48.849694 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:48.849662 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrhx5" Jul 27 20:34:48.849842 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:48.849775 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrhx5" podUID="c3561079-4fcf-46be-8d0d-9a20c39be6d9" Jul 27 20:34:48.849901 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:48.849863 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkd2d" Jul 27 20:34:48.850004 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:48.849984 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkd2d" podUID="05ef8a1e-7bd9-4436-a007-618d151dfd5a" Jul 27 20:34:49.847400 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:49.847360 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j42rs" Jul 27 20:34:49.847889 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:49.847514 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j42rs" podUID="58bd29b9-0422-4c07-90f2-04d39cd07db5" Jul 27 20:34:50.847683 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:50.847650 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrhx5" Jul 27 20:34:50.848130 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:50.847756 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrhx5" podUID="c3561079-4fcf-46be-8d0d-9a20c39be6d9" Jul 27 20:34:50.848130 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:50.847801 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkd2d" Jul 27 20:34:50.848130 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:50.847893 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkd2d" podUID="05ef8a1e-7bd9-4436-a007-618d151dfd5a" Jul 27 20:34:51.847088 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:51.847049 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j42rs" Jul 27 20:34:51.847317 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:51.847197 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j42rs" podUID="58bd29b9-0422-4c07-90f2-04d39cd07db5" Jul 27 20:34:52.848923 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:52.848730 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkd2d" Jul 27 20:34:52.849737 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:52.849012 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkd2d" podUID="05ef8a1e-7bd9-4436-a007-618d151dfd5a" Jul 27 20:34:52.849737 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:52.848730 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrhx5" Jul 27 20:34:52.849737 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:52.849349 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrhx5" podUID="c3561079-4fcf-46be-8d0d-9a20c39be6d9" Jul 27 20:34:52.920705 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:52.920668 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-141.ec2.internal" event={"ID":"27125cf5e1c5c4dd4172f89e5d84d2fd","Type":"ContainerStarted","Data":"181568d42b3fb48bd01e2ac1a49f557b37bcae01b349f0f8e9e295c3056b170b"} Jul 27 20:34:52.922368 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:52.922343 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-141.ec2.internal" event={"ID":"41b9efd1bfa2e292df38ef8f23f5d598","Type":"ContainerStarted","Data":"6969fe9bba47717131e86f77d7b3e426d6bc264d0823fde37493529e345666be"} Jul 27 20:34:52.924619 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:52.924582 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" event={"ID":"e9d570a7-3260-4097-a669-b28a69d3748a","Type":"ContainerStarted","Data":"3c95db158b6f7fabc591046e32720a695b7fa426801f54b462be7fe6917f87f3"} Jul 27 20:34:52.926761 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:52.926634 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-22mpg" event={"ID":"f0a531c5-371d-4024-9a40-a542d5cfdeb4","Type":"ContainerStarted","Data":"c81579d104e496afa47c95392a3556669036b01b824495cf6e76c2277d203fea"} Jul 27 20:34:52.929663 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:52.929456 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" event={"ID":"cda0133a-f0f3-4567-bc30-24c8ea984936","Type":"ContainerStarted","Data":"f485a3c81c581fc7eb6dc12832c9b09a2c9621175dd78a462706de508e835878"} Jul 27 20:34:52.929663 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:52.929535 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" event={"ID":"cda0133a-f0f3-4567-bc30-24c8ea984936","Type":"ContainerStarted","Data":"8a459ee2e32150cf2d059fdccd3925f3baf9b08082becc287cd03821e2565833"} Jul 27 20:34:52.929663 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:52.929590 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" event={"ID":"cda0133a-f0f3-4567-bc30-24c8ea984936","Type":"ContainerStarted","Data":"fffcd47555095d78d7c93179f78da78a57bcd3e5f69993a0ef98a58348dc0f8a"} Jul 27 20:34:52.930672 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:52.930649 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vf9q2" event={"ID":"c5f8ec4d-adb1-4938-8887-6708463e4b02","Type":"ContainerStarted","Data":"bb4b116930017e79ea99620c777652fedb9789c5e5071f3b21a016dceac14a65"} Jul 27 20:34:52.932402 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:52.932354 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ljtzv" event={"ID":"35f7c023-fb9f-40ea-9f5f-547b26383bb8","Type":"ContainerStarted","Data":"e82eaa6391260e20916f31ec62ad1b0b3c07b45deb0b38b00ed02ddec98307cf"} Jul 27 20:34:52.933930 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:52.933884 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l6lsr" event={"ID":"c33a8970-0e52-4e5e-82bf-e8823e1f88d9","Type":"ContainerStarted","Data":"fae24d9023966fc9b1b758472f90be2458fd402dd261e65239fe0ae75e5740bd"} Jul 27 20:34:52.935825 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:52.935796 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2gppc" event={"ID":"8c122d10-fe63-4972-87c0-4cf309037cc8","Type":"ContainerStarted","Data":"fa43653162c055a73177c5294bd8fa1789fbced447ad987426f4dbdc28acdb05"} Jul 27 20:34:52.937035 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:52.936992 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-141.ec2.internal" podStartSLOduration=20.936980106 podStartE2EDuration="20.936980106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:34:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:34:52.936558166 +0000 UTC m=+22.672538456" watchObservedRunningTime="2026-07-27 20:34:52.936980106 +0000 UTC m=+22.672960400" Jul 27 20:34:52.937427 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:52.937400 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j899g" event={"ID":"b956291a-cecc-490c-8d57-8c1c36575a89","Type":"ContainerStarted","Data":"7dcd457fc43a4cad3d978dbcbc08499b58cc7a7af2428d93759e86c253ac023d"} Jul 27 20:34:52.951053 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:52.951012 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2gppc" podStartSLOduration=2.466294755 podStartE2EDuration="21.95099812s" totalImagesPullingTime="19.484703365s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:34:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:34:52.950948569 +0000 UTC m=+22.686928856" watchObservedRunningTime="2026-07-27 20:34:52.95099812 +0000 UTC m=+22.686978593" Jul 27 20:34:52.967652 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:52.967578 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-22mpg" podStartSLOduration=2.458240578 podStartE2EDuration="21.96756781s" totalImagesPullingTime="19.509327232s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:34:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:34:52.967089537 +0000 UTC m=+22.703069824" watchObservedRunningTime="2026-07-27 20:34:52.96756781 +0000 UTC m=+22.703548096" Jul 27 20:34:52.982240 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:52.982041 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-l6lsr" podStartSLOduration=10.703868133 podStartE2EDuration="21.982026458s" totalImagesPullingTime="11.278158325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:34:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:34:52.981580153 +0000 UTC m=+22.717560441" watchObservedRunningTime="2026-07-27 20:34:52.982026458 +0000 UTC m=+22.718006745" Jul 27 20:34:53.011631 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:53.011571 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5dp7l" podStartSLOduration=2.549378686 podStartE2EDuration="22.011556081s" totalImagesPullingTime="19.462177395s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:34:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:34:53.011306421 +0000 UTC m=+22.747286707" watchObservedRunningTime="2026-07-27 20:34:53.011556081 +0000 UTC m=+22.747536367" Jul 27 20:34:53.051680 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:53.051628 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vf9q2" podStartSLOduration=10.722156789 podStartE2EDuration="22.051588214s" totalImagesPullingTime="11.329431425s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:34:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:34:53.050973561 +0000 UTC m=+22.786953848" watchObservedRunningTime="2026-07-27 20:34:53.051588214 +0000 UTC m=+22.787568505" Jul 27 20:34:53.743391 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:53.743370 2570 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 27 20:34:53.846831 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:53.846782 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j42rs" Jul 27 20:34:53.847024 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:53.846902 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j42rs" podUID="58bd29b9-0422-4c07-90f2-04d39cd07db5" Jul 27 20:34:53.940977 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:53.940942 2570 generic.go:356] "Generic (PLEG): container finished" podID="35f7c023-fb9f-40ea-9f5f-547b26383bb8" containerID="e82eaa6391260e20916f31ec62ad1b0b3c07b45deb0b38b00ed02ddec98307cf" exitCode=0 Jul 27 20:34:53.941425 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:53.941012 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ljtzv" event={"ID":"35f7c023-fb9f-40ea-9f5f-547b26383bb8","Type":"ContainerDied","Data":"e82eaa6391260e20916f31ec62ad1b0b3c07b45deb0b38b00ed02ddec98307cf"} Jul 27 20:34:53.942533 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:53.942456 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jhjqt" event={"ID":"f782e3d0-da2b-4b64-ac2e-c50230b51a3a","Type":"ContainerStarted","Data":"f5f31c432380169d7e6c9adec1a646488329d4833704dc5d7156007b33872c5f"} Jul 27 20:34:53.944347 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:53.944325 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j899g" event={"ID":"b956291a-cecc-490c-8d57-8c1c36575a89","Type":"ContainerStarted","Data":"69ef31f19398713d9d766122fe2ec3795ef1af62c61b57262605adac478a0eca"} Jul 27 20:34:53.945763 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:53.945731 2570 generic.go:356] "Generic (PLEG): container finished" podID="41b9efd1bfa2e292df38ef8f23f5d598" containerID="6969fe9bba47717131e86f77d7b3e426d6bc264d0823fde37493529e345666be" exitCode=0 Jul 27 20:34:53.945850 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:53.945780 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-141.ec2.internal" event={"ID":"41b9efd1bfa2e292df38ef8f23f5d598","Type":"ContainerDied","Data":"6969fe9bba47717131e86f77d7b3e426d6bc264d0823fde37493529e345666be"} Jul 27 20:34:53.949954 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:53.949435 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" event={"ID":"cda0133a-f0f3-4567-bc30-24c8ea984936","Type":"ContainerStarted","Data":"263da6f99d46db452eedce778003747816814cbe676861dfb879aea419c5133d"} Jul 27 20:34:53.949954 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:53.949516 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" event={"ID":"cda0133a-f0f3-4567-bc30-24c8ea984936","Type":"ContainerStarted","Data":"e29bc01b3654b8b9497fca17412e50d08b05d6499bfcb0c6842223e4acc1d2bb"} Jul 27 20:34:53.949954 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:53.949565 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" event={"ID":"cda0133a-f0f3-4567-bc30-24c8ea984936","Type":"ContainerStarted","Data":"fd5beda7a050dc22535fe18ca87976a360797e5bddca39c0b1e21c7a68a55711"} Jul 27 20:34:53.991752 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:53.991695 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jhjqt" podStartSLOduration=3.533136543 podStartE2EDuration="22.99167218s" totalImagesPullingTime="19.458535637s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:34:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:34:53.990894374 +0000 UTC m=+23.726874661" watchObservedRunningTime="2026-07-27 20:34:53.99167218 +0000 UTC m=+23.727652468" Jul 27 20:34:54.714017 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:54.713892 2570 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T20:34:53.743387606Z","UUID":"cdb13bf9-8d5d-4c27-b43b-0a4cc57fe4de","Handler":null,"Name":"","Endpoint":""} Jul 27 20:34:54.715924 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:54.715785 2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 27 20:34:54.715924 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:54.715835 2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 27 20:34:54.847380 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:54.847292 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkd2d" Jul 27 20:34:54.847380 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:54.847298 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrhx5" Jul 27 20:34:54.847620 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:54.847432 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkd2d" podUID="05ef8a1e-7bd9-4436-a007-618d151dfd5a" Jul 27 20:34:54.847620 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:54.847514 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrhx5" podUID="c3561079-4fcf-46be-8d0d-9a20c39be6d9" Jul 27 20:34:54.953883 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:54.953829 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-141.ec2.internal" event={"ID":"41b9efd1bfa2e292df38ef8f23f5d598","Type":"ContainerStarted","Data":"736fe3c63e61c12ae9489f491bba2b8848f09b8e036847a82a6020bfec4e6c60"} Jul 27 20:34:54.956532 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:54.956503 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j899g" event={"ID":"b956291a-cecc-490c-8d57-8c1c36575a89","Type":"ContainerStarted","Data":"9fa6d9bec5336ba18c244dfc97109d24edc191aa3b0e212c5d142ddc09a7419b"} Jul 27 20:34:54.968853 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:54.968791 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-141.ec2.internal" podStartSLOduration=22.968771649 podStartE2EDuration="22.968771649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:34:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:34:54.968683378 +0000 UTC m=+24.704663664" watchObservedRunningTime="2026-07-27 20:34:54.968771649 +0000 UTC m=+24.704751937" Jul 27 20:34:54.986773 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:54.986720 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j899g" podStartSLOduration=2.98605869 podStartE2EDuration="23.986703623s" totalImagesPullingTime="21.000644933s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:34:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:34:54.986072432 +0000 UTC m=+24.722052718" watchObservedRunningTime="2026-07-27 20:34:54.986703623 +0000 UTC m=+24.722683911" Jul 27 20:34:55.846855 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:55.846820 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j42rs" Jul 27 20:34:55.847057 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:55.846954 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j42rs" podUID="58bd29b9-0422-4c07-90f2-04d39cd07db5" Jul 27 20:34:55.962934 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:55.962899 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" event={"ID":"cda0133a-f0f3-4567-bc30-24c8ea984936","Type":"ContainerStarted","Data":"5d07e8f1c4ba7beb224f3d298eafd441593a821d575b73dedfb805a1429537db"} Jul 27 20:34:56.847049 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:56.847009 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrhx5" Jul 27 20:34:56.847248 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:56.847007 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkd2d" Jul 27 20:34:56.847248 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:56.847143 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrhx5" podUID="c3561079-4fcf-46be-8d0d-9a20c39be6d9" Jul 27 20:34:56.847248 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:56.847239 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkd2d" podUID="05ef8a1e-7bd9-4436-a007-618d151dfd5a" Jul 27 20:34:57.563228 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:57.563155 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-l6lsr" Jul 27 20:34:57.563827 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:57.563807 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-l6lsr" Jul 27 20:34:57.847787 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:57.847638 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j42rs" Jul 27 20:34:57.847886 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:57.847872 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j42rs" podUID="58bd29b9-0422-4c07-90f2-04d39cd07db5" Jul 27 20:34:57.971210 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:57.971174 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" event={"ID":"cda0133a-f0f3-4567-bc30-24c8ea984936","Type":"ContainerStarted","Data":"505ee8f7da85b17e79b6670ba275963d33c577f96fb2e69f6e2aabb8193338fd"} Jul 27 20:34:57.971545 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:57.971477 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-l6lsr" Jul 27 20:34:57.971545 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:57.971528 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:57.971747 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:57.971585 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:57.971747 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:57.971654 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:57.971923 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:57.971902 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l6lsr" Jul 27 20:34:57.987733 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:57.987708 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:57.988175 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:57.988158 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:34:57.996692 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:57.996648 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" podStartSLOduration=7.417480218 podStartE2EDuration="26.996636234s" totalImagesPullingTime="19.579156016s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:34:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:34:57.995053385 +0000 UTC m=+27.731033672" watchObservedRunningTime="2026-07-27 20:34:57.996636234 +0000 UTC m=+27.732616521" Jul 27 20:34:58.847677 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:58.847567 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrhx5" Jul 27 20:34:58.848370 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:58.847568 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkd2d" Jul 27 20:34:58.848370 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:58.847699 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrhx5" podUID="c3561079-4fcf-46be-8d0d-9a20c39be6d9" Jul 27 20:34:58.848370 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:58.847806 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkd2d" podUID="05ef8a1e-7bd9-4436-a007-618d151dfd5a" Jul 27 20:34:58.974701 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:58.974666 2570 generic.go:356] "Generic (PLEG): container finished" podID="35f7c023-fb9f-40ea-9f5f-547b26383bb8" containerID="f678aa4d8a837ba5cfb87faa5ee92ef6f710e4d8d24ef171da19ca850595d47e" exitCode=0 Jul 27 20:34:58.974837 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:58.974747 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ljtzv" event={"ID":"35f7c023-fb9f-40ea-9f5f-547b26383bb8","Type":"ContainerDied","Data":"f678aa4d8a837ba5cfb87faa5ee92ef6f710e4d8d24ef171da19ca850595d47e"} Jul 27 20:34:59.846928 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:34:59.846897 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j42rs" Jul 27 20:34:59.847091 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:34:59.847032 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j42rs" podUID="58bd29b9-0422-4c07-90f2-04d39cd07db5" Jul 27 20:35:00.241588 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:00.241362 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tkd2d"] Jul 27 20:35:00.242052 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:00.241718 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkd2d" Jul 27 20:35:00.242052 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:00.241849 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkd2d" podUID="05ef8a1e-7bd9-4436-a007-618d151dfd5a" Jul 27 20:35:00.242184 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:00.242064 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zrhx5"] Jul 27 20:35:00.242184 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:00.242154 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrhx5" Jul 27 20:35:00.242280 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:00.242255 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrhx5" podUID="c3561079-4fcf-46be-8d0d-9a20c39be6d9" Jul 27 20:35:00.245442 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:00.245416 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j42rs"] Jul 27 20:35:00.245575 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:00.245500 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j42rs" Jul 27 20:35:00.245647 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:00.245588 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j42rs" podUID="58bd29b9-0422-4c07-90f2-04d39cd07db5" Jul 27 20:35:00.980842 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:00.980806 2570 generic.go:356] "Generic (PLEG): container finished" podID="35f7c023-fb9f-40ea-9f5f-547b26383bb8" containerID="88f35857c53710230c1c36907ca5c74bb0a0b034fc80599af78e6b6f26bd7220" exitCode=0 Jul 27 20:35:00.980986 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:00.980854 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ljtzv" event={"ID":"35f7c023-fb9f-40ea-9f5f-547b26383bb8","Type":"ContainerDied","Data":"88f35857c53710230c1c36907ca5c74bb0a0b034fc80599af78e6b6f26bd7220"} Jul 27 20:35:01.847098 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:01.847065 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkd2d" Jul 27 20:35:01.847478 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:01.847105 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j42rs" Jul 27 20:35:01.847478 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:01.847105 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrhx5" Jul 27 20:35:01.847478 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:01.847178 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkd2d" podUID="05ef8a1e-7bd9-4436-a007-618d151dfd5a" Jul 27 20:35:01.847478 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:01.847267 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j42rs" podUID="58bd29b9-0422-4c07-90f2-04d39cd07db5" Jul 27 20:35:01.847478 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:01.847340 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrhx5" podUID="c3561079-4fcf-46be-8d0d-9a20c39be6d9" Jul 27 20:35:02.986895 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:02.986859 2570 generic.go:356] "Generic (PLEG): container finished" podID="35f7c023-fb9f-40ea-9f5f-547b26383bb8" containerID="416d42967e2211d8e8a7b10406f435c99281b7fe8aac92fdc34c6be4ea7beabf" exitCode=0 Jul 27 20:35:02.987239 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:02.986901 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ljtzv" event={"ID":"35f7c023-fb9f-40ea-9f5f-547b26383bb8","Type":"ContainerDied","Data":"416d42967e2211d8e8a7b10406f435c99281b7fe8aac92fdc34c6be4ea7beabf"} Jul 27 20:35:03.847198 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:03.846981 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkd2d" Jul 27 20:35:03.847477 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:03.846982 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrhx5" Jul 27 20:35:03.847477 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:03.847311 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkd2d" podUID="05ef8a1e-7bd9-4436-a007-618d151dfd5a" Jul 27 20:35:03.847477 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:03.847347 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrhx5" podUID="c3561079-4fcf-46be-8d0d-9a20c39be6d9" Jul 27 20:35:03.847477 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:03.846991 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j42rs" Jul 27 20:35:03.847688 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:03.847478 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j42rs" podUID="58bd29b9-0422-4c07-90f2-04d39cd07db5" Jul 27 20:35:04.419922 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:04.419888 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3561079-4fcf-46be-8d0d-9a20c39be6d9-original-pull-secret\") pod \"global-pull-secret-syncer-zrhx5\" (UID: \"c3561079-4fcf-46be-8d0d-9a20c39be6d9\") " pod="kube-system/global-pull-secret-syncer-zrhx5" Jul 27 20:35:04.420338 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:04.420035 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 20:35:04.420338 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:04.420101 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3561079-4fcf-46be-8d0d-9a20c39be6d9-original-pull-secret podName:c3561079-4fcf-46be-8d0d-9a20c39be6d9 nodeName:}" failed. No retries permitted until 2026-07-27 20:35:36.420081616 +0000 UTC m=+66.156061881 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3561079-4fcf-46be-8d0d-9a20c39be6d9-original-pull-secret") pod "global-pull-secret-syncer-zrhx5" (UID: "c3561079-4fcf-46be-8d0d-9a20c39be6d9") : object "kube-system"/"original-pull-secret" not registered Jul 27 20:35:04.520344 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:04.520301 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05ef8a1e-7bd9-4436-a007-618d151dfd5a-metrics-certs\") pod \"network-metrics-daemon-tkd2d\" (UID: \"05ef8a1e-7bd9-4436-a007-618d151dfd5a\") " pod="openshift-multus/network-metrics-daemon-tkd2d" Jul 27 20:35:04.520520 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:04.520401 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 20:35:04.520520 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:04.520469 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05ef8a1e-7bd9-4436-a007-618d151dfd5a-metrics-certs podName:05ef8a1e-7bd9-4436-a007-618d151dfd5a nodeName:}" failed. No retries permitted until 2026-07-27 20:35:36.520448689 +0000 UTC m=+66.256428955 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05ef8a1e-7bd9-4436-a007-618d151dfd5a-metrics-certs") pod "network-metrics-daemon-tkd2d" (UID: "05ef8a1e-7bd9-4436-a007-618d151dfd5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 20:35:04.621839 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:04.621637 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6dcq\" (UniqueName: \"kubernetes.io/projected/58bd29b9-0422-4c07-90f2-04d39cd07db5-kube-api-access-m6dcq\") pod \"network-check-target-j42rs\" (UID: \"58bd29b9-0422-4c07-90f2-04d39cd07db5\") " pod="openshift-network-diagnostics/network-check-target-j42rs" Jul 27 20:35:04.621839 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:04.621788 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 20:35:04.621839 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:04.621811 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 20:35:04.621839 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:04.621821 2570 projected.go:196] Error preparing data for projected volume kube-api-access-m6dcq for pod openshift-network-diagnostics/network-check-target-j42rs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 20:35:04.622131 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:04.621878 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58bd29b9-0422-4c07-90f2-04d39cd07db5-kube-api-access-m6dcq podName:58bd29b9-0422-4c07-90f2-04d39cd07db5 nodeName:}" failed. No retries permitted until 2026-07-27 20:35:36.621864181 +0000 UTC m=+66.357844446 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-m6dcq" (UniqueName: "kubernetes.io/projected/58bd29b9-0422-4c07-90f2-04d39cd07db5-kube-api-access-m6dcq") pod "network-check-target-j42rs" (UID: "58bd29b9-0422-4c07-90f2-04d39cd07db5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 20:35:05.593058 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.593027 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-141.ec2.internal" event="NodeReady" Jul 27 20:35:05.593524 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.593190 2570 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 27 20:35:05.637249 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.637217 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6h8jp"] Jul 27 20:35:05.668876 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.668839 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4xgk6"] Jul 27 20:35:05.669061 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.668983 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6h8jp" Jul 27 20:35:05.671404 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.671376 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 27 20:35:05.671548 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.671438 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 27 20:35:05.671548 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.671529 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-89mzs\"" Jul 27 20:35:05.685489 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.685459 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6h8jp"] Jul 27 20:35:05.685673 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.685501 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4xgk6"] Jul 27 20:35:05.685673 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.685593 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4xgk6" Jul 27 20:35:05.690493 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.689506 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 27 20:35:05.690493 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.689561 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pmzfn\"" Jul 27 20:35:05.690493 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.689883 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 27 20:35:05.690493 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.690273 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 27 20:35:05.831168 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.831126 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c366dc4-9298-4f87-bcc6-9a8794efc1f7-metrics-tls\") pod \"dns-default-6h8jp\" (UID: \"4c366dc4-9298-4f87-bcc6-9a8794efc1f7\") " pod="openshift-dns/dns-default-6h8jp" Jul 27 20:35:05.831348 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.831203 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4c366dc4-9298-4f87-bcc6-9a8794efc1f7-config-volume\") pod \"dns-default-6h8jp\" (UID: \"4c366dc4-9298-4f87-bcc6-9a8794efc1f7\") " pod="openshift-dns/dns-default-6h8jp" Jul 27 20:35:05.831348 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.831244 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtvdz\" (UniqueName: \"kubernetes.io/projected/b2788954-9e92-4c96-b86a-f2efb00df57d-kube-api-access-dtvdz\") pod \"ingress-canary-4xgk6\" (UID: \"b2788954-9e92-4c96-b86a-f2efb00df57d\") " pod="openshift-ingress-canary/ingress-canary-4xgk6" Jul 27 20:35:05.831348 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.831262 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-475lv\" (UniqueName: \"kubernetes.io/projected/4c366dc4-9298-4f87-bcc6-9a8794efc1f7-kube-api-access-475lv\") pod \"dns-default-6h8jp\" (UID: \"4c366dc4-9298-4f87-bcc6-9a8794efc1f7\") " pod="openshift-dns/dns-default-6h8jp" Jul 27 20:35:05.831529 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.831344 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2788954-9e92-4c96-b86a-f2efb00df57d-cert\") pod \"ingress-canary-4xgk6\" (UID: \"b2788954-9e92-4c96-b86a-f2efb00df57d\") " pod="openshift-ingress-canary/ingress-canary-4xgk6" Jul 27 20:35:05.831529 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.831389 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c366dc4-9298-4f87-bcc6-9a8794efc1f7-tmp-dir\") pod \"dns-default-6h8jp\" (UID: \"4c366dc4-9298-4f87-bcc6-9a8794efc1f7\") " pod="openshift-dns/dns-default-6h8jp" Jul 27 20:35:05.847343 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.847258 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkd2d" Jul 27 20:35:05.847509 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.847348 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j42rs" Jul 27 20:35:05.847652 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.847620 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrhx5" Jul 27 20:35:05.850463 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.850289 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 27 20:35:05.850463 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.850319 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 27 20:35:05.850463 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.850335 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 27 20:35:05.850463 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.850377 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 27 20:35:05.850752 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.850618 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5wzbh\"" Jul 27 20:35:05.850843 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.850829 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-z8hwp\"" Jul 27 20:35:05.932692 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.932649 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c366dc4-9298-4f87-bcc6-9a8794efc1f7-metrics-tls\") pod \"dns-default-6h8jp\" (UID: \"4c366dc4-9298-4f87-bcc6-9a8794efc1f7\") " pod="openshift-dns/dns-default-6h8jp" Jul 27 20:35:05.932907 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.932730 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4c366dc4-9298-4f87-bcc6-9a8794efc1f7-config-volume\") pod \"dns-default-6h8jp\" (UID: \"4c366dc4-9298-4f87-bcc6-9a8794efc1f7\") " pod="openshift-dns/dns-default-6h8jp" Jul 27 20:35:05.932907 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.932770 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtvdz\" (UniqueName: \"kubernetes.io/projected/b2788954-9e92-4c96-b86a-f2efb00df57d-kube-api-access-dtvdz\") pod \"ingress-canary-4xgk6\" (UID: \"b2788954-9e92-4c96-b86a-f2efb00df57d\") " pod="openshift-ingress-canary/ingress-canary-4xgk6" Jul 27 20:35:05.932907 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:05.932784 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 20:35:05.932907 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:05.932890 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c366dc4-9298-4f87-bcc6-9a8794efc1f7-metrics-tls podName:4c366dc4-9298-4f87-bcc6-9a8794efc1f7 nodeName:}" failed. No retries permitted until 2026-07-27 20:35:06.432863534 +0000 UTC m=+36.168843822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c366dc4-9298-4f87-bcc6-9a8794efc1f7-metrics-tls") pod "dns-default-6h8jp" (UID: "4c366dc4-9298-4f87-bcc6-9a8794efc1f7") : secret "dns-default-metrics-tls" not found Jul 27 20:35:05.933138 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.932789 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-475lv\" (UniqueName: \"kubernetes.io/projected/4c366dc4-9298-4f87-bcc6-9a8794efc1f7-kube-api-access-475lv\") pod \"dns-default-6h8jp\" (UID: \"4c366dc4-9298-4f87-bcc6-9a8794efc1f7\") " pod="openshift-dns/dns-default-6h8jp" Jul 27 20:35:05.933138 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.932997 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2788954-9e92-4c96-b86a-f2efb00df57d-cert\") pod \"ingress-canary-4xgk6\" (UID: \"b2788954-9e92-4c96-b86a-f2efb00df57d\") " pod="openshift-ingress-canary/ingress-canary-4xgk6" Jul 27 20:35:05.933138 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.933031 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c366dc4-9298-4f87-bcc6-9a8794efc1f7-tmp-dir\") pod \"dns-default-6h8jp\" (UID: \"4c366dc4-9298-4f87-bcc6-9a8794efc1f7\") " pod="openshift-dns/dns-default-6h8jp" Jul 27 20:35:05.933378 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.933358 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c366dc4-9298-4f87-bcc6-9a8794efc1f7-tmp-dir\") pod \"dns-default-6h8jp\" (UID: \"4c366dc4-9298-4f87-bcc6-9a8794efc1f7\") " pod="openshift-dns/dns-default-6h8jp" Jul 27 20:35:05.933442 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.933402 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4c366dc4-9298-4f87-bcc6-9a8794efc1f7-config-volume\") pod \"dns-default-6h8jp\" (UID: \"4c366dc4-9298-4f87-bcc6-9a8794efc1f7\") " pod="openshift-dns/dns-default-6h8jp" Jul 27 20:35:05.937826 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.937805 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2788954-9e92-4c96-b86a-f2efb00df57d-cert\") pod \"ingress-canary-4xgk6\" (UID: \"b2788954-9e92-4c96-b86a-f2efb00df57d\") " pod="openshift-ingress-canary/ingress-canary-4xgk6" Jul 27 20:35:05.942255 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.942229 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtvdz\" (UniqueName: \"kubernetes.io/projected/b2788954-9e92-4c96-b86a-f2efb00df57d-kube-api-access-dtvdz\") pod \"ingress-canary-4xgk6\" (UID: \"b2788954-9e92-4c96-b86a-f2efb00df57d\") " pod="openshift-ingress-canary/ingress-canary-4xgk6" Jul 27 20:35:05.943707 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.943685 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-475lv\" (UniqueName: \"kubernetes.io/projected/4c366dc4-9298-4f87-bcc6-9a8794efc1f7-kube-api-access-475lv\") pod \"dns-default-6h8jp\" (UID: \"4c366dc4-9298-4f87-bcc6-9a8794efc1f7\") " pod="openshift-dns/dns-default-6h8jp" Jul 27 20:35:05.999268 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:05.999235 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4xgk6" Jul 27 20:35:06.437578 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:06.437523 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c366dc4-9298-4f87-bcc6-9a8794efc1f7-metrics-tls\") pod \"dns-default-6h8jp\" (UID: \"4c366dc4-9298-4f87-bcc6-9a8794efc1f7\") " pod="openshift-dns/dns-default-6h8jp" Jul 27 20:35:06.437774 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:06.437697 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 20:35:06.437826 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:06.437773 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c366dc4-9298-4f87-bcc6-9a8794efc1f7-metrics-tls podName:4c366dc4-9298-4f87-bcc6-9a8794efc1f7 nodeName:}" failed. No retries permitted until 2026-07-27 20:35:07.43775023 +0000 UTC m=+37.173730498 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c366dc4-9298-4f87-bcc6-9a8794efc1f7-metrics-tls") pod "dns-default-6h8jp" (UID: "4c366dc4-9298-4f87-bcc6-9a8794efc1f7") : secret "dns-default-metrics-tls" not found Jul 27 20:35:07.201729 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:07.201693 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4xgk6"] Jul 27 20:35:07.286112 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:35:07.286079 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2788954_9e92_4c96_b86a_f2efb00df57d.slice/crio-1a3876659b76b6c20d32aa031dbb9a059c81bb7ba05dfadaa21db52a7e3fc864 WatchSource:0}: Error finding container 1a3876659b76b6c20d32aa031dbb9a059c81bb7ba05dfadaa21db52a7e3fc864: Status 404 returned error can't find the container with id 1a3876659b76b6c20d32aa031dbb9a059c81bb7ba05dfadaa21db52a7e3fc864 Jul 27 20:35:07.446182 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:07.445988 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c366dc4-9298-4f87-bcc6-9a8794efc1f7-metrics-tls\") pod \"dns-default-6h8jp\" (UID: \"4c366dc4-9298-4f87-bcc6-9a8794efc1f7\") " pod="openshift-dns/dns-default-6h8jp" Jul 27 20:35:07.446253 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:07.446143 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 20:35:07.446308 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:07.446298 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c366dc4-9298-4f87-bcc6-9a8794efc1f7-metrics-tls podName:4c366dc4-9298-4f87-bcc6-9a8794efc1f7 nodeName:}" failed. No retries permitted until 2026-07-27 20:35:09.446281427 +0000 UTC m=+39.182261692 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c366dc4-9298-4f87-bcc6-9a8794efc1f7-metrics-tls") pod "dns-default-6h8jp" (UID: "4c366dc4-9298-4f87-bcc6-9a8794efc1f7") : secret "dns-default-metrics-tls" not found Jul 27 20:35:08.001984 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:08.001949 2570 generic.go:356] "Generic (PLEG): container finished" podID="35f7c023-fb9f-40ea-9f5f-547b26383bb8" containerID="c71ad2d56d8d9729cb7fd3e06667ab935600de4fc166e4a4470a281e77a6a8b0" exitCode=0 Jul 27 20:35:08.002176 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:08.002052 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ljtzv" event={"ID":"35f7c023-fb9f-40ea-9f5f-547b26383bb8","Type":"ContainerDied","Data":"c71ad2d56d8d9729cb7fd3e06667ab935600de4fc166e4a4470a281e77a6a8b0"} Jul 27 20:35:08.003746 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:08.003719 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4xgk6" event={"ID":"b2788954-9e92-4c96-b86a-f2efb00df57d","Type":"ContainerStarted","Data":"1a3876659b76b6c20d32aa031dbb9a059c81bb7ba05dfadaa21db52a7e3fc864"} Jul 27 20:35:08.593271 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:08.593227 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-j4g8t"] Jul 27 20:35:08.606244 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:08.606207 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-j4g8t"] Jul 27 20:35:08.606463 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:08.606338 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-j4g8t" Jul 27 20:35:08.609142 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:08.609116 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 27 20:35:08.609142 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:08.609131 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 27 20:35:08.609319 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:08.609194 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 27 20:35:08.610215 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:08.610190 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 27 20:35:08.610379 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:08.610363 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-84dpl\"" Jul 27 20:35:08.757953 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:08.757907 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/63ec3928-279d-40bd-a949-8b39d0345274-signing-key\") pod \"service-ca-658d7b7c78-j4g8t\" (UID: \"63ec3928-279d-40bd-a949-8b39d0345274\") " pod="openshift-service-ca/service-ca-658d7b7c78-j4g8t" Jul 27 20:35:08.758136 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:08.758001 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/63ec3928-279d-40bd-a949-8b39d0345274-signing-cabundle\") pod \"service-ca-658d7b7c78-j4g8t\" (UID: \"63ec3928-279d-40bd-a949-8b39d0345274\") " pod="openshift-service-ca/service-ca-658d7b7c78-j4g8t" Jul 27 20:35:08.758136 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:08.758042 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfw2x\" (UniqueName: \"kubernetes.io/projected/63ec3928-279d-40bd-a949-8b39d0345274-kube-api-access-rfw2x\") pod \"service-ca-658d7b7c78-j4g8t\" (UID: \"63ec3928-279d-40bd-a949-8b39d0345274\") " pod="openshift-service-ca/service-ca-658d7b7c78-j4g8t" Jul 27 20:35:08.859513 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:08.859426 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/63ec3928-279d-40bd-a949-8b39d0345274-signing-key\") pod \"service-ca-658d7b7c78-j4g8t\" (UID: \"63ec3928-279d-40bd-a949-8b39d0345274\") " pod="openshift-service-ca/service-ca-658d7b7c78-j4g8t" Jul 27 20:35:08.859513 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:08.859488 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/63ec3928-279d-40bd-a949-8b39d0345274-signing-cabundle\") pod \"service-ca-658d7b7c78-j4g8t\" (UID: \"63ec3928-279d-40bd-a949-8b39d0345274\") " pod="openshift-service-ca/service-ca-658d7b7c78-j4g8t" Jul 27 20:35:08.860191 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:08.860168 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfw2x\" (UniqueName: \"kubernetes.io/projected/63ec3928-279d-40bd-a949-8b39d0345274-kube-api-access-rfw2x\") pod \"service-ca-658d7b7c78-j4g8t\" (UID: \"63ec3928-279d-40bd-a949-8b39d0345274\") " pod="openshift-service-ca/service-ca-658d7b7c78-j4g8t" Jul 27 20:35:08.860329 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:08.860249 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/63ec3928-279d-40bd-a949-8b39d0345274-signing-cabundle\") pod \"service-ca-658d7b7c78-j4g8t\" (UID: \"63ec3928-279d-40bd-a949-8b39d0345274\") " pod="openshift-service-ca/service-ca-658d7b7c78-j4g8t" Jul 27 20:35:08.864229 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:08.864204 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/63ec3928-279d-40bd-a949-8b39d0345274-signing-key\") pod \"service-ca-658d7b7c78-j4g8t\" (UID: \"63ec3928-279d-40bd-a949-8b39d0345274\") " pod="openshift-service-ca/service-ca-658d7b7c78-j4g8t" Jul 27 20:35:08.869764 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:08.869693 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfw2x\" (UniqueName: \"kubernetes.io/projected/63ec3928-279d-40bd-a949-8b39d0345274-kube-api-access-rfw2x\") pod \"service-ca-658d7b7c78-j4g8t\" (UID: \"63ec3928-279d-40bd-a949-8b39d0345274\") " pod="openshift-service-ca/service-ca-658d7b7c78-j4g8t" Jul 27 20:35:08.918594 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:08.918554 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-j4g8t" Jul 27 20:35:09.010195 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:09.010156 2570 generic.go:356] "Generic (PLEG): container finished" podID="35f7c023-fb9f-40ea-9f5f-547b26383bb8" containerID="71b6b4e31acce2d4c9c4e825ebd384522a95939ef1ae419930d59997e5b69939" exitCode=0 Jul 27 20:35:09.010195 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:09.010204 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ljtzv" event={"ID":"35f7c023-fb9f-40ea-9f5f-547b26383bb8","Type":"ContainerDied","Data":"71b6b4e31acce2d4c9c4e825ebd384522a95939ef1ae419930d59997e5b69939"} Jul 27 20:35:09.464711 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:09.464666 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c366dc4-9298-4f87-bcc6-9a8794efc1f7-metrics-tls\") pod \"dns-default-6h8jp\" (UID: \"4c366dc4-9298-4f87-bcc6-9a8794efc1f7\") " pod="openshift-dns/dns-default-6h8jp" Jul 27 20:35:09.464912 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:09.464825 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 20:35:09.464912 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:09.464905 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c366dc4-9298-4f87-bcc6-9a8794efc1f7-metrics-tls podName:4c366dc4-9298-4f87-bcc6-9a8794efc1f7 nodeName:}" failed. No retries permitted until 2026-07-27 20:35:13.464886634 +0000 UTC m=+43.200866899 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c366dc4-9298-4f87-bcc6-9a8794efc1f7-metrics-tls") pod "dns-default-6h8jp" (UID: "4c366dc4-9298-4f87-bcc6-9a8794efc1f7") : secret "dns-default-metrics-tls" not found Jul 27 20:35:09.609002 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:09.608808 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-j4g8t"] Jul 27 20:35:09.613146 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:35:09.613116 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63ec3928_279d_40bd_a949_8b39d0345274.slice/crio-93654476010e7250115e4f091a38e3163059b26afbab5cf711d6a1763178841a WatchSource:0}: Error finding container 93654476010e7250115e4f091a38e3163059b26afbab5cf711d6a1763178841a: Status 404 returned error can't find the container with id 93654476010e7250115e4f091a38e3163059b26afbab5cf711d6a1763178841a Jul 27 20:35:10.014958 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:10.014919 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4xgk6" event={"ID":"b2788954-9e92-4c96-b86a-f2efb00df57d","Type":"ContainerStarted","Data":"7dbdbb8982064c07ac9ef0efb135c59c04cc6cb8fca6cb0dbcde7bdacf633cc9"} Jul 27 20:35:10.019136 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:10.019104 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ljtzv" event={"ID":"35f7c023-fb9f-40ea-9f5f-547b26383bb8","Type":"ContainerStarted","Data":"6bbabb23ed227d720a27ddb10f16dcebc4c94d0408fc766561225994137f9d2a"} Jul 27 20:35:10.020255 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:10.020231 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-j4g8t" event={"ID":"63ec3928-279d-40bd-a949-8b39d0345274","Type":"ContainerStarted","Data":"93654476010e7250115e4f091a38e3163059b26afbab5cf711d6a1763178841a"} Jul 27 20:35:10.031288 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:10.031233 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4xgk6" podStartSLOduration=2.4542036769999998 podStartE2EDuration="5.031216519s" totalImagesPullingTime="2.577012842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:35:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:35:10.030250806 +0000 UTC m=+39.766231092" watchObservedRunningTime="2026-07-27 20:35:10.031216519 +0000 UTC m=+39.767196805" Jul 27 20:35:10.057127 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:10.057063 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ljtzv" podStartSLOduration=17.481634338 podStartE2EDuration="39.057045904s" totalImagesPullingTime="21.575411566s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:34:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:35:10.055239082 +0000 UTC m=+39.791219381" watchObservedRunningTime="2026-07-27 20:35:10.057045904 +0000 UTC m=+39.793026214" Jul 27 20:35:12.027225 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:12.027189 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-j4g8t" event={"ID":"63ec3928-279d-40bd-a949-8b39d0345274","Type":"ContainerStarted","Data":"2576b9e572db53f16a78048dcfeb483ebf13db6be12af4dc20e429b2c5006f66"} Jul 27 20:35:12.042555 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:12.042380 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-j4g8t" podStartSLOduration=2.092413672 podStartE2EDuration="4.042365988s" totalImagesPullingTime="1.949952316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:35:12.041895581 +0000 UTC m=+41.777875889" watchObservedRunningTime="2026-07-27 20:35:12.042365988 +0000 UTC m=+41.778346274" Jul 27 20:35:13.497505 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:13.497455 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c366dc4-9298-4f87-bcc6-9a8794efc1f7-metrics-tls\") pod \"dns-default-6h8jp\" (UID: \"4c366dc4-9298-4f87-bcc6-9a8794efc1f7\") " pod="openshift-dns/dns-default-6h8jp" Jul 27 20:35:13.498031 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:13.497635 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 20:35:13.498031 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:13.497711 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c366dc4-9298-4f87-bcc6-9a8794efc1f7-metrics-tls podName:4c366dc4-9298-4f87-bcc6-9a8794efc1f7 nodeName:}" failed. No retries permitted until 2026-07-27 20:35:21.497693361 +0000 UTC m=+51.233673626 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c366dc4-9298-4f87-bcc6-9a8794efc1f7-metrics-tls") pod "dns-default-6h8jp" (UID: "4c366dc4-9298-4f87-bcc6-9a8794efc1f7") : secret "dns-default-metrics-tls" not found Jul 27 20:35:21.558040 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:21.557997 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c366dc4-9298-4f87-bcc6-9a8794efc1f7-metrics-tls\") pod \"dns-default-6h8jp\" (UID: \"4c366dc4-9298-4f87-bcc6-9a8794efc1f7\") " pod="openshift-dns/dns-default-6h8jp" Jul 27 20:35:21.561501 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:21.561473 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c366dc4-9298-4f87-bcc6-9a8794efc1f7-metrics-tls\") pod \"dns-default-6h8jp\" (UID: \"4c366dc4-9298-4f87-bcc6-9a8794efc1f7\") " pod="openshift-dns/dns-default-6h8jp" Jul 27 20:35:21.580365 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:21.580322 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6h8jp" Jul 27 20:35:21.743430 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:21.743394 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6h8jp"] Jul 27 20:35:21.747884 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:35:21.747831 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c366dc4_9298_4f87_bcc6_9a8794efc1f7.slice/crio-e051240fe627405cb8ca19735238367e9ef5ff5af214efe1c5c8ce9da8f407d8 WatchSource:0}: Error finding container e051240fe627405cb8ca19735238367e9ef5ff5af214efe1c5c8ce9da8f407d8: Status 404 returned error can't find the container with id e051240fe627405cb8ca19735238367e9ef5ff5af214efe1c5c8ce9da8f407d8 Jul 27 20:35:22.053717 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:22.053676 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6h8jp" event={"ID":"4c366dc4-9298-4f87-bcc6-9a8794efc1f7","Type":"ContainerStarted","Data":"e051240fe627405cb8ca19735238367e9ef5ff5af214efe1c5c8ce9da8f407d8"} Jul 27 20:35:25.062927 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:25.062892 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6h8jp" event={"ID":"4c366dc4-9298-4f87-bcc6-9a8794efc1f7","Type":"ContainerStarted","Data":"d18d0fc37d9be0b06e87bb6cd516f0b6999838e0390c17fed0f7a92daf006bd2"} Jul 27 20:35:25.063308 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:25.062943 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6h8jp" event={"ID":"4c366dc4-9298-4f87-bcc6-9a8794efc1f7","Type":"ContainerStarted","Data":"8751b06f364ce5e06c334360a1b3ff7fce3e0a9437dd86e86a0f0a4c24a96ab2"} Jul 27 20:35:25.063308 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:25.063035 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6h8jp" Jul 27 20:35:25.079970 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:25.079904 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6h8jp" podStartSLOduration=17.447179325 podStartE2EDuration="20.079886618s" totalImagesPullingTime="2.632707293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:35:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:35:25.079515873 +0000 UTC m=+54.815496173" watchObservedRunningTime="2026-07-27 20:35:25.079886618 +0000 UTC m=+54.815866906" Jul 27 20:35:28.715226 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.715182 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-gsmf9"] Jul 27 20:35:28.751899 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.750878 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-6s67c"] Jul 27 20:35:28.751899 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.750934 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-gsmf9" Jul 27 20:35:28.755044 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.755010 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 27 20:35:28.755179 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.755073 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 27 20:35:28.755316 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.755296 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 27 20:35:28.755390 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.755334 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-dgfvr\"" Jul 27 20:35:28.767918 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.766876 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-6s67c"] Jul 27 20:35:28.767918 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.766914 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-gsmf9"] Jul 27 20:35:28.767918 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.766943 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d8b8f8b4-trk4k"] Jul 27 20:35:28.768299 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.768264 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-6s67c" Jul 27 20:35:28.774361 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.774243 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-rgk7p\"" Jul 27 20:35:28.777867 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.777217 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 27 20:35:28.777867 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.777455 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 27 20:35:28.777867 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.777709 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 27 20:35:28.777867 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.777803 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 27 20:35:28.785012 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.784992 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d8b8f8b4-trk4k"] Jul 27 20:35:28.785132 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.785070 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d8b8f8b4-trk4k" Jul 27 20:35:28.787397 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.787376 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 27 20:35:28.787512 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.787404 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 27 20:35:28.787512 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.787433 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 27 20:35:28.787512 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.787465 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 27 20:35:28.787714 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.787699 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-vtzwp\"" Jul 27 20:35:28.802520 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.802495 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/72a84315-ba3a-4bd1-8922-e384b47004c1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-gsmf9\" (UID: \"72a84315-ba3a-4bd1-8922-e384b47004c1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-gsmf9" Jul 27 20:35:28.802685 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.802531 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr5mt\" (UniqueName: \"kubernetes.io/projected/72a84315-ba3a-4bd1-8922-e384b47004c1-kube-api-access-tr5mt\") pod \"cluster-samples-operator-6748489656-gsmf9\" (UID: \"72a84315-ba3a-4bd1-8922-e384b47004c1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-gsmf9" Jul 27 20:35:28.802685 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.802574 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/151eada8-db02-4937-a9fd-25fbc88c8f20-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-6s67c\" (UID: \"151eada8-db02-4937-a9fd-25fbc88c8f20\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-6s67c" Jul 27 20:35:28.802685 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.802621 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47j5r\" (UniqueName: \"kubernetes.io/projected/151eada8-db02-4937-a9fd-25fbc88c8f20-kube-api-access-47j5r\") pod \"cluster-monitoring-operator-548b489cb8-6s67c\" (UID: \"151eada8-db02-4937-a9fd-25fbc88c8f20\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-6s67c" Jul 27 20:35:28.802873 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.802698 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwzpb\" (UniqueName: \"kubernetes.io/projected/9d5ccffd-8b3b-45cb-a4d5-c856fff6bb12-kube-api-access-zwzpb\") pod \"managed-serviceaccount-addon-agent-5d8b8f8b4-trk4k\" (UID: \"9d5ccffd-8b3b-45cb-a4d5-c856fff6bb12\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d8b8f8b4-trk4k" Jul 27 20:35:28.802873 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.802730 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9d5ccffd-8b3b-45cb-a4d5-c856fff6bb12-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5d8b8f8b4-trk4k\" (UID: \"9d5ccffd-8b3b-45cb-a4d5-c856fff6bb12\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d8b8f8b4-trk4k" Jul 27 20:35:28.802873 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.802753 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/151eada8-db02-4937-a9fd-25fbc88c8f20-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-6s67c\" (UID: \"151eada8-db02-4937-a9fd-25fbc88c8f20\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-6s67c" Jul 27 20:35:28.851019 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.850991 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mnsh8"] Jul 27 20:35:28.870927 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.870893 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-67fb5bfb49-zx4sr"] Jul 27 20:35:28.871110 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.871039 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mnsh8" Jul 27 20:35:28.874050 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.873911 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-kch9w\"" Jul 27 20:35:28.874050 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.873963 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 27 20:35:28.874675 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.874334 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 27 20:35:28.884949 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.884925 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-rfmfv"] Jul 27 20:35:28.885065 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.885048 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67fb5bfb49-zx4sr" Jul 27 20:35:28.887452 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.887431 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 27 20:35:28.887569 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.887456 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hj2k5\"" Jul 27 20:35:28.887569 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.887431 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 27 20:35:28.888037 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.888023 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 27 20:35:28.893592 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.893493 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 27 20:35:28.901740 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.901154 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mnsh8"] Jul 27 20:35:28.901740 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.901205 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-rfmfv"] Jul 27 20:35:28.901740 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.901227 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67fb5bfb49-zx4sr"] Jul 27 20:35:28.901740 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.901254 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67fb5bfb49-zx4sr"] Jul 27 20:35:28.901740 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:28.901566 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-lsfmj registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-lsfmj registry-certificates registry-tls trusted-ca]: context canceled" pod="openshift-image-registry/image-registry-67fb5bfb49-zx4sr" podUID="bbc8b410-0fa6-4377-9737-cba844cc7266" Jul 27 20:35:28.901740 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.901694 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-rfmfv" Jul 27 20:35:28.903107 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.903083 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbc8b410-0fa6-4377-9737-cba844cc7266-registry-tls\") pod \"image-registry-67fb5bfb49-zx4sr\" (UID: \"bbc8b410-0fa6-4377-9737-cba844cc7266\") " pod="openshift-image-registry/image-registry-67fb5bfb49-zx4sr" Jul 27 20:35:28.903199 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.903124 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bbc8b410-0fa6-4377-9737-cba844cc7266-image-registry-private-configuration\") pod \"image-registry-67fb5bfb49-zx4sr\" (UID: \"bbc8b410-0fa6-4377-9737-cba844cc7266\") " pod="openshift-image-registry/image-registry-67fb5bfb49-zx4sr" Jul 27 20:35:28.903199 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.903147 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bbc8b410-0fa6-4377-9737-cba844cc7266-registry-certificates\") pod \"image-registry-67fb5bfb49-zx4sr\" (UID: \"bbc8b410-0fa6-4377-9737-cba844cc7266\") " pod="openshift-image-registry/image-registry-67fb5bfb49-zx4sr" Jul 27 20:35:28.903199 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.903163 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4scc\" (UniqueName: \"kubernetes.io/projected/41483315-8990-4cb6-9312-1c14c9476301-kube-api-access-m4scc\") pod \"console-operator-584dcb6b54-rfmfv\" (UID: \"41483315-8990-4cb6-9312-1c14c9476301\") " pod="openshift-console-operator/console-operator-584dcb6b54-rfmfv" Jul 27 20:35:28.903309 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.903202 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsfmj\" (UniqueName: \"kubernetes.io/projected/bbc8b410-0fa6-4377-9737-cba844cc7266-kube-api-access-lsfmj\") pod \"image-registry-67fb5bfb49-zx4sr\" (UID: \"bbc8b410-0fa6-4377-9737-cba844cc7266\") " pod="openshift-image-registry/image-registry-67fb5bfb49-zx4sr" Jul 27 20:35:28.903309 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.903242 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9d5ccffd-8b3b-45cb-a4d5-c856fff6bb12-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5d8b8f8b4-trk4k\" (UID: \"9d5ccffd-8b3b-45cb-a4d5-c856fff6bb12\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d8b8f8b4-trk4k" Jul 27 20:35:28.903373 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.903309 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/151eada8-db02-4937-a9fd-25fbc88c8f20-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-6s67c\" (UID: \"151eada8-db02-4937-a9fd-25fbc88c8f20\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-6s67c" Jul 27 20:35:28.903373 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.903348 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bbc8b410-0fa6-4377-9737-cba844cc7266-installation-pull-secrets\") pod \"image-registry-67fb5bfb49-zx4sr\" (UID: \"bbc8b410-0fa6-4377-9737-cba844cc7266\") " pod="openshift-image-registry/image-registry-67fb5bfb49-zx4sr" Jul 27 20:35:28.903486 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.903468 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6ct6\" (UniqueName: \"kubernetes.io/projected/3a68f100-f17d-4f48-b78e-27ff219c64ac-kube-api-access-q6ct6\") pod \"volume-data-source-validator-549c6d54cf-mnsh8\" (UID: \"3a68f100-f17d-4f48-b78e-27ff219c64ac\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mnsh8" Jul 27 20:35:28.903544 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.903504 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/41483315-8990-4cb6-9312-1c14c9476301-serving-cert\") pod \"console-operator-584dcb6b54-rfmfv\" (UID: \"41483315-8990-4cb6-9312-1c14c9476301\") " pod="openshift-console-operator/console-operator-584dcb6b54-rfmfv" Jul 27 20:35:28.903624 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.903541 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/41483315-8990-4cb6-9312-1c14c9476301-trusted-ca\") pod \"console-operator-584dcb6b54-rfmfv\" (UID: \"41483315-8990-4cb6-9312-1c14c9476301\") " pod="openshift-console-operator/console-operator-584dcb6b54-rfmfv" Jul 27 20:35:28.903624 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.903574 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/151eada8-db02-4937-a9fd-25fbc88c8f20-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-6s67c\" (UID: \"151eada8-db02-4937-a9fd-25fbc88c8f20\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-6s67c" Jul 27 20:35:28.903727 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.903660 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bbc8b410-0fa6-4377-9737-cba844cc7266-bound-sa-token\") pod \"image-registry-67fb5bfb49-zx4sr\" (UID: \"bbc8b410-0fa6-4377-9737-cba844cc7266\") " pod="openshift-image-registry/image-registry-67fb5bfb49-zx4sr" Jul 27 20:35:28.903727 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.903692 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47j5r\" (UniqueName: \"kubernetes.io/projected/151eada8-db02-4937-a9fd-25fbc88c8f20-kube-api-access-47j5r\") pod \"cluster-monitoring-operator-548b489cb8-6s67c\" (UID: \"151eada8-db02-4937-a9fd-25fbc88c8f20\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-6s67c" Jul 27 20:35:28.903727 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.903718 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bbc8b410-0fa6-4377-9737-cba844cc7266-ca-trust-extracted\") pod \"image-registry-67fb5bfb49-zx4sr\" (UID: \"bbc8b410-0fa6-4377-9737-cba844cc7266\") " pod="openshift-image-registry/image-registry-67fb5bfb49-zx4sr" Jul 27 20:35:28.903876 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.903751 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bbc8b410-0fa6-4377-9737-cba844cc7266-trusted-ca\") pod \"image-registry-67fb5bfb49-zx4sr\" (UID: \"bbc8b410-0fa6-4377-9737-cba844cc7266\") " pod="openshift-image-registry/image-registry-67fb5bfb49-zx4sr" Jul 27 20:35:28.904101 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.904078 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwzpb\" (UniqueName: \"kubernetes.io/projected/9d5ccffd-8b3b-45cb-a4d5-c856fff6bb12-kube-api-access-zwzpb\") pod \"managed-serviceaccount-addon-agent-5d8b8f8b4-trk4k\" (UID: \"9d5ccffd-8b3b-45cb-a4d5-c856fff6bb12\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d8b8f8b4-trk4k" Jul 27 20:35:28.904181 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.904127 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/72a84315-ba3a-4bd1-8922-e384b47004c1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-gsmf9\" (UID: \"72a84315-ba3a-4bd1-8922-e384b47004c1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-gsmf9" Jul 27 20:35:28.904241 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.904194 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/41483315-8990-4cb6-9312-1c14c9476301-config\") pod \"console-operator-584dcb6b54-rfmfv\" (UID: \"41483315-8990-4cb6-9312-1c14c9476301\") " pod="openshift-console-operator/console-operator-584dcb6b54-rfmfv" Jul 27 20:35:28.904241 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.904229 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr5mt\" (UniqueName: \"kubernetes.io/projected/72a84315-ba3a-4bd1-8922-e384b47004c1-kube-api-access-tr5mt\") pod \"cluster-samples-operator-6748489656-gsmf9\" (UID: \"72a84315-ba3a-4bd1-8922-e384b47004c1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-gsmf9" Jul 27 20:35:28.904342 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.904298 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/151eada8-db02-4937-a9fd-25fbc88c8f20-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-6s67c\" (UID: \"151eada8-db02-4937-a9fd-25fbc88c8f20\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-6s67c" Jul 27 20:35:28.905404 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.905385 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 27 20:35:28.906075 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.906045 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/151eada8-db02-4937-a9fd-25fbc88c8f20-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-6s67c\" (UID: \"151eada8-db02-4937-a9fd-25fbc88c8f20\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-6s67c" Jul 27 20:35:28.906412 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.906390 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9d5ccffd-8b3b-45cb-a4d5-c856fff6bb12-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5d8b8f8b4-trk4k\" (UID: \"9d5ccffd-8b3b-45cb-a4d5-c856fff6bb12\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d8b8f8b4-trk4k" Jul 27 20:35:28.906774 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.906758 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/72a84315-ba3a-4bd1-8922-e384b47004c1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-gsmf9\" (UID: \"72a84315-ba3a-4bd1-8922-e384b47004c1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-gsmf9" Jul 27 20:35:28.907737 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.907721 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-ss676\"" Jul 27 20:35:28.909197 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.909178 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 27 20:35:28.909285 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.909269 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 27 20:35:28.919770 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.919747 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47j5r\" (UniqueName: \"kubernetes.io/projected/151eada8-db02-4937-a9fd-25fbc88c8f20-kube-api-access-47j5r\") pod \"cluster-monitoring-operator-548b489cb8-6s67c\" (UID: \"151eada8-db02-4937-a9fd-25fbc88c8f20\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-6s67c" Jul 27 20:35:28.933754 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.933714 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 27 20:35:28.935252 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.935231 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwzpb\" (UniqueName: \"kubernetes.io/projected/9d5ccffd-8b3b-45cb-a4d5-c856fff6bb12-kube-api-access-zwzpb\") pod \"managed-serviceaccount-addon-agent-5d8b8f8b4-trk4k\" (UID: \"9d5ccffd-8b3b-45cb-a4d5-c856fff6bb12\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d8b8f8b4-trk4k" Jul 27 20:35:28.935356 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.935327 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr5mt\" (UniqueName: \"kubernetes.io/projected/72a84315-ba3a-4bd1-8922-e384b47004c1-kube-api-access-tr5mt\") pod \"cluster-samples-operator-6748489656-gsmf9\" (UID: \"72a84315-ba3a-4bd1-8922-e384b47004c1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-gsmf9" Jul 27 20:35:28.939735 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:28.939712 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 27 20:35:29.004649 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.004547 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/41483315-8990-4cb6-9312-1c14c9476301-config\") pod \"console-operator-584dcb6b54-rfmfv\" (UID: \"41483315-8990-4cb6-9312-1c14c9476301\") " pod="openshift-console-operator/console-operator-584dcb6b54-rfmfv" Jul 27 20:35:29.004649 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.004586 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbc8b410-0fa6-4377-9737-cba844cc7266-registry-tls\") pod \"image-registry-67fb5bfb49-zx4sr\" (UID: \"bbc8b410-0fa6-4377-9737-cba844cc7266\") " pod="openshift-image-registry/image-registry-67fb5bfb49-zx4sr" Jul 27 20:35:29.004649 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.004628 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bbc8b410-0fa6-4377-9737-cba844cc7266-image-registry-private-configuration\") pod \"image-registry-67fb5bfb49-zx4sr\" (UID: \"bbc8b410-0fa6-4377-9737-cba844cc7266\") " pod="openshift-image-registry/image-registry-67fb5bfb49-zx4sr" Jul 27 20:35:29.004649 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.004647 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bbc8b410-0fa6-4377-9737-cba844cc7266-registry-certificates\") pod \"image-registry-67fb5bfb49-zx4sr\" (UID: \"bbc8b410-0fa6-4377-9737-cba844cc7266\") " pod="openshift-image-registry/image-registry-67fb5bfb49-zx4sr" Jul 27 20:35:29.004951 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.004664 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4scc\" (UniqueName: \"kubernetes.io/projected/41483315-8990-4cb6-9312-1c14c9476301-kube-api-access-m4scc\") pod \"console-operator-584dcb6b54-rfmfv\" (UID: \"41483315-8990-4cb6-9312-1c14c9476301\") " pod="openshift-console-operator/console-operator-584dcb6b54-rfmfv" Jul 27 20:35:29.004951 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.004693 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsfmj\" (UniqueName: \"kubernetes.io/projected/bbc8b410-0fa6-4377-9737-cba844cc7266-kube-api-access-lsfmj\") pod \"image-registry-67fb5bfb49-zx4sr\" (UID: \"bbc8b410-0fa6-4377-9737-cba844cc7266\") " pod="openshift-image-registry/image-registry-67fb5bfb49-zx4sr" Jul 27 20:35:29.004951 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.004719 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bbc8b410-0fa6-4377-9737-cba844cc7266-installation-pull-secrets\") pod \"image-registry-67fb5bfb49-zx4sr\" (UID: \"bbc8b410-0fa6-4377-9737-cba844cc7266\") " pod="openshift-image-registry/image-registry-67fb5bfb49-zx4sr" Jul 27 20:35:29.005186 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.005143 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6ct6\" (UniqueName: \"kubernetes.io/projected/3a68f100-f17d-4f48-b78e-27ff219c64ac-kube-api-access-q6ct6\") pod \"volume-data-source-validator-549c6d54cf-mnsh8\" (UID: \"3a68f100-f17d-4f48-b78e-27ff219c64ac\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mnsh8" Jul 27 20:35:29.005297 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.005195 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/41483315-8990-4cb6-9312-1c14c9476301-serving-cert\") pod \"console-operator-584dcb6b54-rfmfv\" (UID: \"41483315-8990-4cb6-9312-1c14c9476301\") " pod="openshift-console-operator/console-operator-584dcb6b54-rfmfv" Jul 27 20:35:29.005297 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.005233 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/41483315-8990-4cb6-9312-1c14c9476301-trusted-ca\") pod \"console-operator-584dcb6b54-rfmfv\" (UID: \"41483315-8990-4cb6-9312-1c14c9476301\") " pod="openshift-console-operator/console-operator-584dcb6b54-rfmfv" Jul 27 20:35:29.005297 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.005263 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bbc8b410-0fa6-4377-9737-cba844cc7266-bound-sa-token\") pod \"image-registry-67fb5bfb49-zx4sr\" (UID: \"bbc8b410-0fa6-4377-9737-cba844cc7266\") " pod="openshift-image-registry/image-registry-67fb5bfb49-zx4sr" Jul 27 20:35:29.005297 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.005296 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bbc8b410-0fa6-4377-9737-cba844cc7266-ca-trust-extracted\") pod \"image-registry-67fb5bfb49-zx4sr\" (UID: \"bbc8b410-0fa6-4377-9737-cba844cc7266\") " pod="openshift-image-registry/image-registry-67fb5bfb49-zx4sr" Jul 27 20:35:29.005525 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.005319 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bbc8b410-0fa6-4377-9737-cba844cc7266-trusted-ca\") pod \"image-registry-67fb5bfb49-zx4sr\" (UID: \"bbc8b410-0fa6-4377-9737-cba844cc7266\") " pod="openshift-image-registry/image-registry-67fb5bfb49-zx4sr" Jul 27 20:35:29.005660 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.005633 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/41483315-8990-4cb6-9312-1c14c9476301-config\") pod \"console-operator-584dcb6b54-rfmfv\" (UID: \"41483315-8990-4cb6-9312-1c14c9476301\") " pod="openshift-console-operator/console-operator-584dcb6b54-rfmfv" Jul 27 20:35:29.006023 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.005965 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bbc8b410-0fa6-4377-9737-cba844cc7266-registry-certificates\") pod \"image-registry-67fb5bfb49-zx4sr\" (UID: \"bbc8b410-0fa6-4377-9737-cba844cc7266\") " pod="openshift-image-registry/image-registry-67fb5bfb49-zx4sr" Jul 27 20:35:29.006023 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.006010 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bbc8b410-0fa6-4377-9737-cba844cc7266-ca-trust-extracted\") pod \"image-registry-67fb5bfb49-zx4sr\" (UID: \"bbc8b410-0fa6-4377-9737-cba844cc7266\") " pod="openshift-image-registry/image-registry-67fb5bfb49-zx4sr" Jul 27 20:35:29.006589 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.006569 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bbc8b410-0fa6-4377-9737-cba844cc7266-trusted-ca\") pod \"image-registry-67fb5bfb49-zx4sr\" (UID: \"bbc8b410-0fa6-4377-9737-cba844cc7266\") " pod="openshift-image-registry/image-registry-67fb5bfb49-zx4sr" Jul 27 20:35:29.006688 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.006664 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/41483315-8990-4cb6-9312-1c14c9476301-trusted-ca\") pod \"console-operator-584dcb6b54-rfmfv\" (UID: \"41483315-8990-4cb6-9312-1c14c9476301\") " pod="openshift-console-operator/console-operator-584dcb6b54-rfmfv" Jul 27 20:35:29.007532 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.007512 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bbc8b410-0fa6-4377-9737-cba844cc7266-image-registry-private-configuration\") pod \"image-registry-67fb5bfb49-zx4sr\" (UID: \"bbc8b410-0fa6-4377-9737-cba844cc7266\") " pod="openshift-image-registry/image-registry-67fb5bfb49-zx4sr" Jul 27 20:35:29.007981 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.007959 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbc8b410-0fa6-4377-9737-cba844cc7266-registry-tls\") pod \"image-registry-67fb5bfb49-zx4sr\" (UID: \"bbc8b410-0fa6-4377-9737-cba844cc7266\") " pod="openshift-image-registry/image-registry-67fb5bfb49-zx4sr" Jul 27 20:35:29.008072 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.008051 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/41483315-8990-4cb6-9312-1c14c9476301-serving-cert\") pod \"console-operator-584dcb6b54-rfmfv\" (UID: \"41483315-8990-4cb6-9312-1c14c9476301\") " pod="openshift-console-operator/console-operator-584dcb6b54-rfmfv" Jul 27 20:35:29.008116 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.008107 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bbc8b410-0fa6-4377-9737-cba844cc7266-installation-pull-secrets\") pod \"image-registry-67fb5bfb49-zx4sr\" (UID: \"bbc8b410-0fa6-4377-9737-cba844cc7266\") " pod="openshift-image-registry/image-registry-67fb5bfb49-zx4sr" Jul 27 20:35:29.016344 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.016318 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bbc8b410-0fa6-4377-9737-cba844cc7266-bound-sa-token\") pod \"image-registry-67fb5bfb49-zx4sr\" (UID: \"bbc8b410-0fa6-4377-9737-cba844cc7266\") " pod="openshift-image-registry/image-registry-67fb5bfb49-zx4sr" Jul 27 20:35:29.016528 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.016509 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4scc\" (UniqueName: \"kubernetes.io/projected/41483315-8990-4cb6-9312-1c14c9476301-kube-api-access-m4scc\") pod \"console-operator-584dcb6b54-rfmfv\" (UID: \"41483315-8990-4cb6-9312-1c14c9476301\") " pod="openshift-console-operator/console-operator-584dcb6b54-rfmfv" Jul 27 20:35:29.016591 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.016509 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsfmj\" (UniqueName: \"kubernetes.io/projected/bbc8b410-0fa6-4377-9737-cba844cc7266-kube-api-access-lsfmj\") pod \"image-registry-67fb5bfb49-zx4sr\" (UID: \"bbc8b410-0fa6-4377-9737-cba844cc7266\") " pod="openshift-image-registry/image-registry-67fb5bfb49-zx4sr" Jul 27 20:35:29.017062 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.017047 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6ct6\" (UniqueName: \"kubernetes.io/projected/3a68f100-f17d-4f48-b78e-27ff219c64ac-kube-api-access-q6ct6\") pod \"volume-data-source-validator-549c6d54cf-mnsh8\" (UID: \"3a68f100-f17d-4f48-b78e-27ff219c64ac\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mnsh8" Jul 27 20:35:29.067951 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.067919 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-gsmf9" Jul 27 20:35:29.073988 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.073960 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67fb5bfb49-zx4sr" Jul 27 20:35:29.084958 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.084934 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-6s67c" Jul 27 20:35:29.106115 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.106085 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d8b8f8b4-trk4k" Jul 27 20:35:29.123247 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.123221 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67fb5bfb49-zx4sr" Jul 27 20:35:29.182785 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.182750 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mnsh8" Jul 27 20:35:29.207431 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.207397 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bbc8b410-0fa6-4377-9737-cba844cc7266-trusted-ca\") pod \"bbc8b410-0fa6-4377-9737-cba844cc7266\" (UID: \"bbc8b410-0fa6-4377-9737-cba844cc7266\") " Jul 27 20:35:29.207625 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.207448 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bbc8b410-0fa6-4377-9737-cba844cc7266-bound-sa-token\") pod \"bbc8b410-0fa6-4377-9737-cba844cc7266\" (UID: \"bbc8b410-0fa6-4377-9737-cba844cc7266\") " Jul 27 20:35:29.207625 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.207482 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bbc8b410-0fa6-4377-9737-cba844cc7266-image-registry-private-configuration\") pod \"bbc8b410-0fa6-4377-9737-cba844cc7266\" (UID: \"bbc8b410-0fa6-4377-9737-cba844cc7266\") " Jul 27 20:35:29.207625 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.207523 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bbc8b410-0fa6-4377-9737-cba844cc7266-installation-pull-secrets\") pod \"bbc8b410-0fa6-4377-9737-cba844cc7266\" (UID: \"bbc8b410-0fa6-4377-9737-cba844cc7266\") " Jul 27 20:35:29.207625 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.207562 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bbc8b410-0fa6-4377-9737-cba844cc7266-registry-certificates\") pod \"bbc8b410-0fa6-4377-9737-cba844cc7266\" (UID: \"bbc8b410-0fa6-4377-9737-cba844cc7266\") " Jul 27 20:35:29.207625 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.207613 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsfmj\" (UniqueName: \"kubernetes.io/projected/bbc8b410-0fa6-4377-9737-cba844cc7266-kube-api-access-lsfmj\") pod \"bbc8b410-0fa6-4377-9737-cba844cc7266\" (UID: \"bbc8b410-0fa6-4377-9737-cba844cc7266\") " Jul 27 20:35:29.207886 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.207652 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bbc8b410-0fa6-4377-9737-cba844cc7266-ca-trust-extracted\") pod \"bbc8b410-0fa6-4377-9737-cba844cc7266\" (UID: \"bbc8b410-0fa6-4377-9737-cba844cc7266\") " Jul 27 20:35:29.207886 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.207697 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbc8b410-0fa6-4377-9737-cba844cc7266-registry-tls\") pod \"bbc8b410-0fa6-4377-9737-cba844cc7266\" (UID: \"bbc8b410-0fa6-4377-9737-cba844cc7266\") " Jul 27 20:35:29.208109 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.208080 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbc8b410-0fa6-4377-9737-cba844cc7266-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "bbc8b410-0fa6-4377-9737-cba844cc7266" (UID: "bbc8b410-0fa6-4377-9737-cba844cc7266"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:35:29.208589 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.208540 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc8b410-0fa6-4377-9737-cba844cc7266-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "bbc8b410-0fa6-4377-9737-cba844cc7266" (UID: "bbc8b410-0fa6-4377-9737-cba844cc7266"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 20:35:29.209120 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.209069 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbc8b410-0fa6-4377-9737-cba844cc7266-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "bbc8b410-0fa6-4377-9737-cba844cc7266" (UID: "bbc8b410-0fa6-4377-9737-cba844cc7266"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:35:29.211718 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.211674 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc8b410-0fa6-4377-9737-cba844cc7266-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "bbc8b410-0fa6-4377-9737-cba844cc7266" (UID: "bbc8b410-0fa6-4377-9737-cba844cc7266"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:35:29.212820 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.212038 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc8b410-0fa6-4377-9737-cba844cc7266-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "bbc8b410-0fa6-4377-9737-cba844cc7266" (UID: "bbc8b410-0fa6-4377-9737-cba844cc7266"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:35:29.212820 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.212766 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbc8b410-0fa6-4377-9737-cba844cc7266-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "bbc8b410-0fa6-4377-9737-cba844cc7266" (UID: "bbc8b410-0fa6-4377-9737-cba844cc7266"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 20:35:29.214558 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.213050 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc8b410-0fa6-4377-9737-cba844cc7266-kube-api-access-lsfmj" (OuterVolumeSpecName: "kube-api-access-lsfmj") pod "bbc8b410-0fa6-4377-9737-cba844cc7266" (UID: "bbc8b410-0fa6-4377-9737-cba844cc7266"). InnerVolumeSpecName "kube-api-access-lsfmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:35:29.214558 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.213460 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbc8b410-0fa6-4377-9737-cba844cc7266-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "bbc8b410-0fa6-4377-9737-cba844cc7266" (UID: "bbc8b410-0fa6-4377-9737-cba844cc7266"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 20:35:29.221335 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.220844 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-rfmfv" Jul 27 20:35:29.262703 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.262520 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-gsmf9"] Jul 27 20:35:29.308879 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.308844 2570 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bbc8b410-0fa6-4377-9737-cba844cc7266-installation-pull-secrets\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:35:29.308879 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.308879 2570 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bbc8b410-0fa6-4377-9737-cba844cc7266-registry-certificates\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:35:29.309134 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.308895 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsfmj\" (UniqueName: \"kubernetes.io/projected/bbc8b410-0fa6-4377-9737-cba844cc7266-kube-api-access-lsfmj\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:35:29.309134 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.308910 2570 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bbc8b410-0fa6-4377-9737-cba844cc7266-ca-trust-extracted\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:35:29.309134 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.308923 2570 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbc8b410-0fa6-4377-9737-cba844cc7266-registry-tls\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:35:29.309134 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.308936 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bbc8b410-0fa6-4377-9737-cba844cc7266-trusted-ca\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:35:29.309134 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.308948 2570 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bbc8b410-0fa6-4377-9737-cba844cc7266-bound-sa-token\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:35:29.309134 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.308962 2570 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bbc8b410-0fa6-4377-9737-cba844cc7266-image-registry-private-configuration\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:35:29.323796 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.323719 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-6s67c"] Jul 27 20:35:29.330270 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:35:29.330237 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod151eada8_db02_4937_a9fd_25fbc88c8f20.slice/crio-f5ce4e58bce9db2e090c7f799315142796b3066e189973b7a89a035f865fa455 WatchSource:0}: Error finding container f5ce4e58bce9db2e090c7f799315142796b3066e189973b7a89a035f865fa455: Status 404 returned error can't find the container with id f5ce4e58bce9db2e090c7f799315142796b3066e189973b7a89a035f865fa455 Jul 27 20:35:29.336850 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.336813 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d8b8f8b4-trk4k"] Jul 27 20:35:29.377088 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.377058 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mnsh8"] Jul 27 20:35:29.380059 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:35:29.380032 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a68f100_f17d_4f48_b78e_27ff219c64ac.slice/crio-1dbe84cfa15e186204d65974c13a2203e92317ce2d473775c1bc71aec5e695a9 WatchSource:0}: Error finding container 1dbe84cfa15e186204d65974c13a2203e92317ce2d473775c1bc71aec5e695a9: Status 404 returned error can't find the container with id 1dbe84cfa15e186204d65974c13a2203e92317ce2d473775c1bc71aec5e695a9 Jul 27 20:35:29.408204 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:29.408174 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-rfmfv"] Jul 27 20:35:29.410958 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:35:29.410937 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41483315_8990_4cb6_9312_1c14c9476301.slice/crio-0abc515a0ca177f89f9808e5f9ceffd3dd36a07978d09d2c9011b16017a7e557 WatchSource:0}: Error finding container 0abc515a0ca177f89f9808e5f9ceffd3dd36a07978d09d2c9011b16017a7e557: Status 404 returned error can't find the container with id 0abc515a0ca177f89f9808e5f9ceffd3dd36a07978d09d2c9011b16017a7e557 Jul 27 20:35:30.002984 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:30.002796 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ltxxw" Jul 27 20:35:30.080952 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:30.080861 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-6s67c" event={"ID":"151eada8-db02-4937-a9fd-25fbc88c8f20","Type":"ContainerStarted","Data":"f5ce4e58bce9db2e090c7f799315142796b3066e189973b7a89a035f865fa455"} Jul 27 20:35:30.088725 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:30.088541 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-gsmf9" event={"ID":"72a84315-ba3a-4bd1-8922-e384b47004c1","Type":"ContainerStarted","Data":"de455aca74599663ae718411d8df9534b2baa15c7ec0e6a4222d5e0db56fdd31"} Jul 27 20:35:30.098749 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:30.091764 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mnsh8" event={"ID":"3a68f100-f17d-4f48-b78e-27ff219c64ac","Type":"ContainerStarted","Data":"1dbe84cfa15e186204d65974c13a2203e92317ce2d473775c1bc71aec5e695a9"} Jul 27 20:35:30.098749 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:30.093805 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d8b8f8b4-trk4k" event={"ID":"9d5ccffd-8b3b-45cb-a4d5-c856fff6bb12","Type":"ContainerStarted","Data":"c64c0869da3aabbc5fd8d4567620a081b7aa9b16aceef160455326d3ec2a2cf1"} Jul 27 20:35:30.098749 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:30.098097 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67fb5bfb49-zx4sr" Jul 27 20:35:30.103648 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:30.103583 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-rfmfv" event={"ID":"41483315-8990-4cb6-9312-1c14c9476301","Type":"ContainerStarted","Data":"0abc515a0ca177f89f9808e5f9ceffd3dd36a07978d09d2c9011b16017a7e557"} Jul 27 20:35:30.145216 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:30.145175 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67fb5bfb49-zx4sr"] Jul 27 20:35:30.152066 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:30.151979 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-67fb5bfb49-zx4sr"] Jul 27 20:35:30.853178 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:30.853131 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbc8b410-0fa6-4377-9737-cba844cc7266" path="/var/lib/kubelet/pods/bbc8b410-0fa6-4377-9737-cba844cc7266/volumes" Jul 27 20:35:32.108834 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:32.108803 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-s4xkz"] Jul 27 20:35:32.113647 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:32.113621 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s4xkz" Jul 27 20:35:32.116308 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:32.116283 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 27 20:35:32.116448 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:32.116288 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 27 20:35:32.117243 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:32.117217 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 27 20:35:32.117371 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:32.117249 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 27 20:35:32.117371 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:32.117276 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-t5cnl\"" Jul 27 20:35:32.124137 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:32.124114 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s4xkz"] Jul 27 20:35:32.129424 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:32.129400 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-crio-socket\") pod \"insights-runtime-extractor-s4xkz\" (UID: \"a4512db5-a1ae-4484-be6e-cd5b7b3046ee\") " pod="openshift-insights/insights-runtime-extractor-s4xkz" Jul 27 20:35:32.129562 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:32.129434 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-data-volume\") pod \"insights-runtime-extractor-s4xkz\" (UID: \"a4512db5-a1ae-4484-be6e-cd5b7b3046ee\") " pod="openshift-insights/insights-runtime-extractor-s4xkz" Jul 27 20:35:32.129562 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:32.129459 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6dch\" (UniqueName: \"kubernetes.io/projected/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-kube-api-access-c6dch\") pod \"insights-runtime-extractor-s4xkz\" (UID: \"a4512db5-a1ae-4484-be6e-cd5b7b3046ee\") " pod="openshift-insights/insights-runtime-extractor-s4xkz" Jul 27 20:35:32.129562 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:32.129517 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4xkz\" (UID: \"a4512db5-a1ae-4484-be6e-cd5b7b3046ee\") " pod="openshift-insights/insights-runtime-extractor-s4xkz" Jul 27 20:35:32.129562 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:32.129553 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4xkz\" (UID: \"a4512db5-a1ae-4484-be6e-cd5b7b3046ee\") " pod="openshift-insights/insights-runtime-extractor-s4xkz" Jul 27 20:35:32.229956 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:32.229913 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4xkz\" (UID: \"a4512db5-a1ae-4484-be6e-cd5b7b3046ee\") " pod="openshift-insights/insights-runtime-extractor-s4xkz" Jul 27 20:35:32.230146 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:32.229969 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4xkz\" (UID: \"a4512db5-a1ae-4484-be6e-cd5b7b3046ee\") " pod="openshift-insights/insights-runtime-extractor-s4xkz" Jul 27 20:35:32.230146 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:32.230031 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-crio-socket\") pod \"insights-runtime-extractor-s4xkz\" (UID: \"a4512db5-a1ae-4484-be6e-cd5b7b3046ee\") " pod="openshift-insights/insights-runtime-extractor-s4xkz" Jul 27 20:35:32.230146 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:32.230055 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-data-volume\") pod \"insights-runtime-extractor-s4xkz\" (UID: \"a4512db5-a1ae-4484-be6e-cd5b7b3046ee\") " pod="openshift-insights/insights-runtime-extractor-s4xkz" Jul 27 20:35:32.230146 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:32.230063 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 20:35:32.230146 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:32.230082 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6dch\" (UniqueName: \"kubernetes.io/projected/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-kube-api-access-c6dch\") pod \"insights-runtime-extractor-s4xkz\" (UID: \"a4512db5-a1ae-4484-be6e-cd5b7b3046ee\") " pod="openshift-insights/insights-runtime-extractor-s4xkz" Jul 27 20:35:32.230379 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:32.230150 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-crio-socket\") pod \"insights-runtime-extractor-s4xkz\" (UID: \"a4512db5-a1ae-4484-be6e-cd5b7b3046ee\") " pod="openshift-insights/insights-runtime-extractor-s4xkz" Jul 27 20:35:32.230379 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:32.230163 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-kube-rbac-proxy-cm podName:a4512db5-a1ae-4484-be6e-cd5b7b3046ee nodeName:}" failed. No retries permitted until 2026-07-27 20:35:32.730141827 +0000 UTC m=+62.466122092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s4xkz" (UID: "a4512db5-a1ae-4484-be6e-cd5b7b3046ee") : configmap "kube-rbac-proxy" not found Jul 27 20:35:32.230379 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:32.230175 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 20:35:32.230379 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:32.230244 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-insights-runtime-extractor-tls podName:a4512db5-a1ae-4484-be6e-cd5b7b3046ee nodeName:}" failed. No retries permitted until 2026-07-27 20:35:32.730227726 +0000 UTC m=+62.466207991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s4xkz" (UID: "a4512db5-a1ae-4484-be6e-cd5b7b3046ee") : secret "insights-runtime-extractor-tls" not found Jul 27 20:35:32.230509 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:32.230382 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-data-volume\") pod \"insights-runtime-extractor-s4xkz\" (UID: \"a4512db5-a1ae-4484-be6e-cd5b7b3046ee\") " pod="openshift-insights/insights-runtime-extractor-s4xkz" Jul 27 20:35:32.243013 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:32.242975 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6dch\" (UniqueName: \"kubernetes.io/projected/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-kube-api-access-c6dch\") pod \"insights-runtime-extractor-s4xkz\" (UID: \"a4512db5-a1ae-4484-be6e-cd5b7b3046ee\") " pod="openshift-insights/insights-runtime-extractor-s4xkz" Jul 27 20:35:32.733767 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:32.733726 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4xkz\" (UID: \"a4512db5-a1ae-4484-be6e-cd5b7b3046ee\") " pod="openshift-insights/insights-runtime-extractor-s4xkz" Jul 27 20:35:32.733963 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:32.733778 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4xkz\" (UID: \"a4512db5-a1ae-4484-be6e-cd5b7b3046ee\") " pod="openshift-insights/insights-runtime-extractor-s4xkz" Jul 27 20:35:32.733963 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:32.733878 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 20:35:32.733963 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:32.733918 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 20:35:32.733963 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:32.733960 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-kube-rbac-proxy-cm podName:a4512db5-a1ae-4484-be6e-cd5b7b3046ee nodeName:}" failed. No retries permitted until 2026-07-27 20:35:33.733939649 +0000 UTC m=+63.469919922 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s4xkz" (UID: "a4512db5-a1ae-4484-be6e-cd5b7b3046ee") : configmap "kube-rbac-proxy" not found Jul 27 20:35:32.734179 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:32.733980 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-insights-runtime-extractor-tls podName:a4512db5-a1ae-4484-be6e-cd5b7b3046ee nodeName:}" failed. No retries permitted until 2026-07-27 20:35:33.733970717 +0000 UTC m=+63.469950982 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s4xkz" (UID: "a4512db5-a1ae-4484-be6e-cd5b7b3046ee") : secret "insights-runtime-extractor-tls" not found Jul 27 20:35:32.832665 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:32.832634 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6h8jp_4c366dc4-9298-4f87-bcc6-9a8794efc1f7/dns/0.log" Jul 27 20:35:33.012726 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:33.012646 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6h8jp_4c366dc4-9298-4f87-bcc6-9a8794efc1f7/kube-rbac-proxy/0.log" Jul 27 20:35:33.110720 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:33.110639 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-rfmfv" event={"ID":"41483315-8990-4cb6-9312-1c14c9476301","Type":"ContainerStarted","Data":"2fad884045eb406e85763bd1f14458045004478cf1d23e7008fb837e5085a8fc"} Jul 27 20:35:33.111191 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:33.110922 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-rfmfv" Jul 27 20:35:33.112848 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:33.112823 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-6s67c" event={"ID":"151eada8-db02-4937-a9fd-25fbc88c8f20","Type":"ContainerStarted","Data":"19240539b15dbf5f33b9887dc6aa7d5d74531c246038d589f91f05d55b2f30eb"} Jul 27 20:35:33.115374 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:33.115035 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-gsmf9" event={"ID":"72a84315-ba3a-4bd1-8922-e384b47004c1","Type":"ContainerStarted","Data":"2d0b041d201c1f8b39d0651ec1bb46fd9a5017be0aea2b2d2dd3a11718c86a9a"} Jul 27 20:35:33.115374 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:33.115082 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-gsmf9" event={"ID":"72a84315-ba3a-4bd1-8922-e384b47004c1","Type":"ContainerStarted","Data":"c035075fedb671221730d6f65f71f73d829bd513a9406c9f480631f86fd3eab2"} Jul 27 20:35:33.116769 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:33.116743 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mnsh8" event={"ID":"3a68f100-f17d-4f48-b78e-27ff219c64ac","Type":"ContainerStarted","Data":"8a514d57e8d67d651c3791922910d2aaa3e1e5cb8723137d4e47582ec0aefd4d"} Jul 27 20:35:33.129253 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:33.129014 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-rfmfv" podStartSLOduration=2.10292829 podStartE2EDuration="5.1289906s" totalImagesPullingTime="3.02606231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:35:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:35:33.12882711 +0000 UTC m=+62.864807424" watchObservedRunningTime="2026-07-27 20:35:33.1289906 +0000 UTC m=+62.864970891" Jul 27 20:35:33.147084 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:33.147021 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mnsh8" podStartSLOduration=2.094236488 podStartE2EDuration="5.146999646s" totalImagesPullingTime="3.052763158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:35:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:35:33.146436699 +0000 UTC m=+62.882416987" watchObservedRunningTime="2026-07-27 20:35:33.146999646 +0000 UTC m=+62.882979934" Jul 27 20:35:33.188447 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:33.187325 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-6s67c" podStartSLOduration=2.084450072 podStartE2EDuration="5.187303504s" totalImagesPullingTime="3.102853432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:35:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:35:33.187029063 +0000 UTC m=+62.923009373" watchObservedRunningTime="2026-07-27 20:35:33.187303504 +0000 UTC m=+62.923283794" Jul 27 20:35:33.188447 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:33.187975 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-gsmf9" podStartSLOduration=2.086888829 podStartE2EDuration="5.187958329s" totalImagesPullingTime="3.1010695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:35:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:35:33.166517797 +0000 UTC m=+62.902498087" watchObservedRunningTime="2026-07-27 20:35:33.187958329 +0000 UTC m=+62.923938617" Jul 27 20:35:33.262663 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:33.262634 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-rfmfv" Jul 27 20:35:33.742388 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:33.742348 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4xkz\" (UID: \"a4512db5-a1ae-4484-be6e-cd5b7b3046ee\") " pod="openshift-insights/insights-runtime-extractor-s4xkz" Jul 27 20:35:33.742388 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:33.742394 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4xkz\" (UID: \"a4512db5-a1ae-4484-be6e-cd5b7b3046ee\") " pod="openshift-insights/insights-runtime-extractor-s4xkz" Jul 27 20:35:33.742661 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:33.742470 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 20:35:33.742661 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:33.742498 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 20:35:33.742661 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:33.742532 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-kube-rbac-proxy-cm podName:a4512db5-a1ae-4484-be6e-cd5b7b3046ee nodeName:}" failed. No retries permitted until 2026-07-27 20:35:35.742516239 +0000 UTC m=+65.478496505 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s4xkz" (UID: "a4512db5-a1ae-4484-be6e-cd5b7b3046ee") : configmap "kube-rbac-proxy" not found Jul 27 20:35:33.742661 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:33.742578 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-insights-runtime-extractor-tls podName:a4512db5-a1ae-4484-be6e-cd5b7b3046ee nodeName:}" failed. No retries permitted until 2026-07-27 20:35:35.742562912 +0000 UTC m=+65.478543178 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s4xkz" (UID: "a4512db5-a1ae-4484-be6e-cd5b7b3046ee") : secret "insights-runtime-extractor-tls" not found Jul 27 20:35:34.011222 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:34.011142 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2gppc_8c122d10-fe63-4972-87c0-4cf309037cc8/dns-node-resolver/0.log" Jul 27 20:35:35.067948 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:35.067910 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6h8jp" Jul 27 20:35:35.410545 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:35.410457 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vf9q2_c5f8ec4d-adb1-4938-8887-6708463e4b02/node-ca/0.log" Jul 27 20:35:35.759559 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:35.759511 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4xkz\" (UID: \"a4512db5-a1ae-4484-be6e-cd5b7b3046ee\") " pod="openshift-insights/insights-runtime-extractor-s4xkz" Jul 27 20:35:35.759811 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:35.759649 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4xkz\" (UID: \"a4512db5-a1ae-4484-be6e-cd5b7b3046ee\") " pod="openshift-insights/insights-runtime-extractor-s4xkz" Jul 27 20:35:35.759811 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:35.759701 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 20:35:35.759811 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:35.759727 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 20:35:35.759811 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:35.759779 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-kube-rbac-proxy-cm podName:a4512db5-a1ae-4484-be6e-cd5b7b3046ee nodeName:}" failed. No retries permitted until 2026-07-27 20:35:39.759762634 +0000 UTC m=+69.495742898 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s4xkz" (UID: "a4512db5-a1ae-4484-be6e-cd5b7b3046ee") : configmap "kube-rbac-proxy" not found Jul 27 20:35:35.759811 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:35.759792 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-insights-runtime-extractor-tls podName:a4512db5-a1ae-4484-be6e-cd5b7b3046ee nodeName:}" failed. No retries permitted until 2026-07-27 20:35:39.759785685 +0000 UTC m=+69.495765949 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s4xkz" (UID: "a4512db5-a1ae-4484-be6e-cd5b7b3046ee") : secret "insights-runtime-extractor-tls" not found Jul 27 20:35:35.810849 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:35.810819 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4xgk6_b2788954-9e92-4c96-b86a-f2efb00df57d/serve-healthcheck-canary/0.log" Jul 27 20:35:36.463849 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:36.463813 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3561079-4fcf-46be-8d0d-9a20c39be6d9-original-pull-secret\") pod \"global-pull-secret-syncer-zrhx5\" (UID: \"c3561079-4fcf-46be-8d0d-9a20c39be6d9\") " pod="kube-system/global-pull-secret-syncer-zrhx5" Jul 27 20:35:36.466997 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:36.466979 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 27 20:35:36.476810 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:36.476783 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3561079-4fcf-46be-8d0d-9a20c39be6d9-original-pull-secret\") pod \"global-pull-secret-syncer-zrhx5\" (UID: \"c3561079-4fcf-46be-8d0d-9a20c39be6d9\") " pod="kube-system/global-pull-secret-syncer-zrhx5" Jul 27 20:35:36.564260 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:36.564222 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05ef8a1e-7bd9-4436-a007-618d151dfd5a-metrics-certs\") pod \"network-metrics-daemon-tkd2d\" (UID: \"05ef8a1e-7bd9-4436-a007-618d151dfd5a\") " pod="openshift-multus/network-metrics-daemon-tkd2d" Jul 27 20:35:36.566943 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:36.566925 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 27 20:35:36.577355 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:36.577326 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05ef8a1e-7bd9-4436-a007-618d151dfd5a-metrics-certs\") pod \"network-metrics-daemon-tkd2d\" (UID: \"05ef8a1e-7bd9-4436-a007-618d151dfd5a\") " pod="openshift-multus/network-metrics-daemon-tkd2d" Jul 27 20:35:36.665188 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:36.665147 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6dcq\" (UniqueName: \"kubernetes.io/projected/58bd29b9-0422-4c07-90f2-04d39cd07db5-kube-api-access-m6dcq\") pod \"network-check-target-j42rs\" (UID: \"58bd29b9-0422-4c07-90f2-04d39cd07db5\") " pod="openshift-network-diagnostics/network-check-target-j42rs" Jul 27 20:35:36.667927 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:36.667910 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 27 20:35:36.678303 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:36.678280 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 27 20:35:36.689156 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:36.689122 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6dcq\" (UniqueName: \"kubernetes.io/projected/58bd29b9-0422-4c07-90f2-04d39cd07db5-kube-api-access-m6dcq\") pod \"network-check-target-j42rs\" (UID: \"58bd29b9-0422-4c07-90f2-04d39cd07db5\") " pod="openshift-network-diagnostics/network-check-target-j42rs" Jul 27 20:35:36.761007 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:36.760924 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5wzbh\"" Jul 27 20:35:36.768231 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:36.768207 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-z8hwp\"" Jul 27 20:35:36.769222 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:36.769206 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkd2d" Jul 27 20:35:36.770995 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:36.770976 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrhx5" Jul 27 20:35:36.776678 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:36.776655 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j42rs" Jul 27 20:35:36.924572 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:36.924538 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j42rs"] Jul 27 20:35:36.929085 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:35:36.929053 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58bd29b9_0422_4c07_90f2_04d39cd07db5.slice/crio-66f4f3406b4f611200f09f6f382858097617f8d175b98e5bbc86825ac8066c1d WatchSource:0}: Error finding container 66f4f3406b4f611200f09f6f382858097617f8d175b98e5bbc86825ac8066c1d: Status 404 returned error can't find the container with id 66f4f3406b4f611200f09f6f382858097617f8d175b98e5bbc86825ac8066c1d Jul 27 20:35:36.934167 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:36.934142 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zrhx5"] Jul 27 20:35:36.953491 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:36.953460 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tkd2d"] Jul 27 20:35:36.956250 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:35:36.956224 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05ef8a1e_7bd9_4436_a007_618d151dfd5a.slice/crio-3469b2428cef6bc8025d9641549cfae6f73b0943ef8217045e407905f6862032 WatchSource:0}: Error finding container 3469b2428cef6bc8025d9641549cfae6f73b0943ef8217045e407905f6862032: Status 404 returned error can't find the container with id 3469b2428cef6bc8025d9641549cfae6f73b0943ef8217045e407905f6862032 Jul 27 20:35:37.134397 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:37.134300 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zrhx5" event={"ID":"c3561079-4fcf-46be-8d0d-9a20c39be6d9","Type":"ContainerStarted","Data":"90f30e5d0b38f2239ee633aef6b86e151521c85b80cbd3b86164272be9135e53"} Jul 27 20:35:37.135381 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:37.135357 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j42rs" event={"ID":"58bd29b9-0422-4c07-90f2-04d39cd07db5","Type":"ContainerStarted","Data":"66f4f3406b4f611200f09f6f382858097617f8d175b98e5bbc86825ac8066c1d"} Jul 27 20:35:37.136476 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:37.136451 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tkd2d" event={"ID":"05ef8a1e-7bd9-4436-a007-618d151dfd5a","Type":"ContainerStarted","Data":"3469b2428cef6bc8025d9641549cfae6f73b0943ef8217045e407905f6862032"} Jul 27 20:35:38.064259 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:38.063228 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-6b59f"] Jul 27 20:35:38.066833 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:38.065470 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-6b59f" Jul 27 20:35:38.071222 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:38.071002 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 27 20:35:38.071222 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:38.071002 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 27 20:35:38.072131 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:38.072111 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 27 20:35:38.072539 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:38.072343 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9thvq\"" Jul 27 20:35:38.082259 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:38.075556 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-6b59f"] Jul 27 20:35:38.145949 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:38.145918 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tkd2d" event={"ID":"05ef8a1e-7bd9-4436-a007-618d151dfd5a","Type":"ContainerStarted","Data":"ce56ca2bc34499e019c0a81c15e57a05f967be6d37312719a775fd8d0fe957b2"} Jul 27 20:35:38.185992 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:38.185945 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c79f7f0b-c4e0-424f-9067-10f0ffe0963b-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-6b59f\" (UID: \"c79f7f0b-c4e0-424f-9067-10f0ffe0963b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6b59f" Jul 27 20:35:38.186180 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:38.186042 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c79f7f0b-c4e0-424f-9067-10f0ffe0963b-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-6b59f\" (UID: \"c79f7f0b-c4e0-424f-9067-10f0ffe0963b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6b59f" Jul 27 20:35:38.186180 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:38.186074 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwd5q\" (UniqueName: \"kubernetes.io/projected/c79f7f0b-c4e0-424f-9067-10f0ffe0963b-kube-api-access-gwd5q\") pod \"prometheus-operator-84d69f794d-6b59f\" (UID: \"c79f7f0b-c4e0-424f-9067-10f0ffe0963b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6b59f" Jul 27 20:35:38.186180 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:38.186109 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c79f7f0b-c4e0-424f-9067-10f0ffe0963b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-6b59f\" (UID: \"c79f7f0b-c4e0-424f-9067-10f0ffe0963b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6b59f" Jul 27 20:35:38.287188 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:38.287145 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c79f7f0b-c4e0-424f-9067-10f0ffe0963b-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-6b59f\" (UID: \"c79f7f0b-c4e0-424f-9067-10f0ffe0963b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6b59f" Jul 27 20:35:38.287388 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:38.287202 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwd5q\" (UniqueName: \"kubernetes.io/projected/c79f7f0b-c4e0-424f-9067-10f0ffe0963b-kube-api-access-gwd5q\") pod \"prometheus-operator-84d69f794d-6b59f\" (UID: \"c79f7f0b-c4e0-424f-9067-10f0ffe0963b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6b59f" Jul 27 20:35:38.287388 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:38.287237 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c79f7f0b-c4e0-424f-9067-10f0ffe0963b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-6b59f\" (UID: \"c79f7f0b-c4e0-424f-9067-10f0ffe0963b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6b59f" Jul 27 20:35:38.287388 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:38.287312 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c79f7f0b-c4e0-424f-9067-10f0ffe0963b-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-6b59f\" (UID: \"c79f7f0b-c4e0-424f-9067-10f0ffe0963b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6b59f" Jul 27 20:35:38.288409 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:38.288385 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c79f7f0b-c4e0-424f-9067-10f0ffe0963b-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-6b59f\" (UID: \"c79f7f0b-c4e0-424f-9067-10f0ffe0963b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6b59f" Jul 27 20:35:38.290167 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:38.290147 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c79f7f0b-c4e0-424f-9067-10f0ffe0963b-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-6b59f\" (UID: \"c79f7f0b-c4e0-424f-9067-10f0ffe0963b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6b59f" Jul 27 20:35:38.290398 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:38.290375 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c79f7f0b-c4e0-424f-9067-10f0ffe0963b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-6b59f\" (UID: \"c79f7f0b-c4e0-424f-9067-10f0ffe0963b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6b59f" Jul 27 20:35:38.300806 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:38.300781 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwd5q\" (UniqueName: \"kubernetes.io/projected/c79f7f0b-c4e0-424f-9067-10f0ffe0963b-kube-api-access-gwd5q\") pod \"prometheus-operator-84d69f794d-6b59f\" (UID: \"c79f7f0b-c4e0-424f-9067-10f0ffe0963b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6b59f" Jul 27 20:35:38.393969 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:38.393932 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-6b59f" Jul 27 20:35:38.553871 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:38.553800 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-6b59f"] Jul 27 20:35:38.556899 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:35:38.556868 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc79f7f0b_c4e0_424f_9067_10f0ffe0963b.slice/crio-03b9991e31cf3e5b94b2928a5053c9d1db8ce8e097126e999fc21437bf3cdd52 WatchSource:0}: Error finding container 03b9991e31cf3e5b94b2928a5053c9d1db8ce8e097126e999fc21437bf3cdd52: Status 404 returned error can't find the container with id 03b9991e31cf3e5b94b2928a5053c9d1db8ce8e097126e999fc21437bf3cdd52 Jul 27 20:35:39.153337 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:39.153299 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tkd2d" event={"ID":"05ef8a1e-7bd9-4436-a007-618d151dfd5a","Type":"ContainerStarted","Data":"c4e55f51625fbe34b6cad2caff0e24d4dd5c6f4327e5e07999bdc8d555c5ef9b"} Jul 27 20:35:39.154818 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:39.154772 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-6b59f" event={"ID":"c79f7f0b-c4e0-424f-9067-10f0ffe0963b","Type":"ContainerStarted","Data":"03b9991e31cf3e5b94b2928a5053c9d1db8ce8e097126e999fc21437bf3cdd52"} Jul 27 20:35:39.169635 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:39.169567 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tkd2d" podStartSLOduration=67.177762769 podStartE2EDuration="1m8.169547529s" totalImagesPullingTime="991.78476ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:34:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:35:39.169251979 +0000 UTC m=+68.905232267" watchObservedRunningTime="2026-07-27 20:35:39.169547529 +0000 UTC m=+68.905527819" Jul 27 20:35:39.799173 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:39.798581 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4xkz\" (UID: \"a4512db5-a1ae-4484-be6e-cd5b7b3046ee\") " pod="openshift-insights/insights-runtime-extractor-s4xkz" Jul 27 20:35:39.799173 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:39.798669 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4xkz\" (UID: \"a4512db5-a1ae-4484-be6e-cd5b7b3046ee\") " pod="openshift-insights/insights-runtime-extractor-s4xkz" Jul 27 20:35:39.799173 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:39.798783 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 20:35:39.799173 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:39.798845 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-insights-runtime-extractor-tls podName:a4512db5-a1ae-4484-be6e-cd5b7b3046ee nodeName:}" failed. No retries permitted until 2026-07-27 20:35:47.79882572 +0000 UTC m=+77.534806011 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s4xkz" (UID: "a4512db5-a1ae-4484-be6e-cd5b7b3046ee") : secret "insights-runtime-extractor-tls" not found Jul 27 20:35:39.799173 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:39.799088 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 20:35:39.799173 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:39.799140 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-kube-rbac-proxy-cm podName:a4512db5-a1ae-4484-be6e-cd5b7b3046ee nodeName:}" failed. No retries permitted until 2026-07-27 20:35:47.799124073 +0000 UTC m=+77.535104339 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s4xkz" (UID: "a4512db5-a1ae-4484-be6e-cd5b7b3046ee") : configmap "kube-rbac-proxy" not found Jul 27 20:35:42.954898 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:42.954864 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-96695bc7-4k645"] Jul 27 20:35:42.957473 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:42.957449 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-96695bc7-4k645" Jul 27 20:35:42.960095 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:42.960037 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 27 20:35:42.960234 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:42.960130 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 27 20:35:42.960234 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:42.960211 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 27 20:35:42.960363 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:42.960211 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rmhbg\"" Jul 27 20:35:42.961024 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:42.961006 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 27 20:35:42.961180 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:42.961027 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 27 20:35:42.961180 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:42.961118 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 27 20:35:42.961323 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:42.961268 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 27 20:35:42.968420 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:42.968396 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-96695bc7-4k645"] Jul 27 20:35:43.024868 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:43.024824 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf78p\" (UniqueName: \"kubernetes.io/projected/d90ced29-8d27-4cd9-9ec1-a6e7132a1365-kube-api-access-mf78p\") pod \"console-96695bc7-4k645\" (UID: \"d90ced29-8d27-4cd9-9ec1-a6e7132a1365\") " pod="openshift-console/console-96695bc7-4k645" Jul 27 20:35:43.025053 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:43.024934 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d90ced29-8d27-4cd9-9ec1-a6e7132a1365-console-config\") pod \"console-96695bc7-4k645\" (UID: \"d90ced29-8d27-4cd9-9ec1-a6e7132a1365\") " pod="openshift-console/console-96695bc7-4k645" Jul 27 20:35:43.025053 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:43.025030 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d90ced29-8d27-4cd9-9ec1-a6e7132a1365-console-oauth-config\") pod \"console-96695bc7-4k645\" (UID: \"d90ced29-8d27-4cd9-9ec1-a6e7132a1365\") " pod="openshift-console/console-96695bc7-4k645" Jul 27 20:35:43.025154 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:43.025069 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d90ced29-8d27-4cd9-9ec1-a6e7132a1365-oauth-serving-cert\") pod \"console-96695bc7-4k645\" (UID: \"d90ced29-8d27-4cd9-9ec1-a6e7132a1365\") " pod="openshift-console/console-96695bc7-4k645" Jul 27 20:35:43.025154 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:43.025118 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d90ced29-8d27-4cd9-9ec1-a6e7132a1365-service-ca\") pod \"console-96695bc7-4k645\" (UID: \"d90ced29-8d27-4cd9-9ec1-a6e7132a1365\") " pod="openshift-console/console-96695bc7-4k645" Jul 27 20:35:43.025242 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:43.025156 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d90ced29-8d27-4cd9-9ec1-a6e7132a1365-console-serving-cert\") pod \"console-96695bc7-4k645\" (UID: \"d90ced29-8d27-4cd9-9ec1-a6e7132a1365\") " pod="openshift-console/console-96695bc7-4k645" Jul 27 20:35:43.126077 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:43.126044 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d90ced29-8d27-4cd9-9ec1-a6e7132a1365-oauth-serving-cert\") pod \"console-96695bc7-4k645\" (UID: \"d90ced29-8d27-4cd9-9ec1-a6e7132a1365\") " pod="openshift-console/console-96695bc7-4k645" Jul 27 20:35:43.126277 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:43.126110 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d90ced29-8d27-4cd9-9ec1-a6e7132a1365-service-ca\") pod \"console-96695bc7-4k645\" (UID: \"d90ced29-8d27-4cd9-9ec1-a6e7132a1365\") " pod="openshift-console/console-96695bc7-4k645" Jul 27 20:35:43.126277 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:43.126135 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d90ced29-8d27-4cd9-9ec1-a6e7132a1365-console-serving-cert\") pod \"console-96695bc7-4k645\" (UID: \"d90ced29-8d27-4cd9-9ec1-a6e7132a1365\") " pod="openshift-console/console-96695bc7-4k645" Jul 27 20:35:43.126277 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:43.126154 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf78p\" (UniqueName: \"kubernetes.io/projected/d90ced29-8d27-4cd9-9ec1-a6e7132a1365-kube-api-access-mf78p\") pod \"console-96695bc7-4k645\" (UID: \"d90ced29-8d27-4cd9-9ec1-a6e7132a1365\") " pod="openshift-console/console-96695bc7-4k645" Jul 27 20:35:43.126277 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:43.126198 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d90ced29-8d27-4cd9-9ec1-a6e7132a1365-console-config\") pod \"console-96695bc7-4k645\" (UID: \"d90ced29-8d27-4cd9-9ec1-a6e7132a1365\") " pod="openshift-console/console-96695bc7-4k645" Jul 27 20:35:43.126277 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:43.126241 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d90ced29-8d27-4cd9-9ec1-a6e7132a1365-console-oauth-config\") pod \"console-96695bc7-4k645\" (UID: \"d90ced29-8d27-4cd9-9ec1-a6e7132a1365\") " pod="openshift-console/console-96695bc7-4k645" Jul 27 20:35:43.126892 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:43.126869 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d90ced29-8d27-4cd9-9ec1-a6e7132a1365-oauth-serving-cert\") pod \"console-96695bc7-4k645\" (UID: \"d90ced29-8d27-4cd9-9ec1-a6e7132a1365\") " pod="openshift-console/console-96695bc7-4k645" Jul 27 20:35:43.126892 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:43.126889 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d90ced29-8d27-4cd9-9ec1-a6e7132a1365-service-ca\") pod \"console-96695bc7-4k645\" (UID: \"d90ced29-8d27-4cd9-9ec1-a6e7132a1365\") " pod="openshift-console/console-96695bc7-4k645" Jul 27 20:35:43.127455 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:43.127412 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d90ced29-8d27-4cd9-9ec1-a6e7132a1365-console-config\") pod \"console-96695bc7-4k645\" (UID: \"d90ced29-8d27-4cd9-9ec1-a6e7132a1365\") " pod="openshift-console/console-96695bc7-4k645" Jul 27 20:35:43.129232 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:43.129202 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d90ced29-8d27-4cd9-9ec1-a6e7132a1365-console-oauth-config\") pod \"console-96695bc7-4k645\" (UID: \"d90ced29-8d27-4cd9-9ec1-a6e7132a1365\") " pod="openshift-console/console-96695bc7-4k645" Jul 27 20:35:43.129776 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:43.129756 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d90ced29-8d27-4cd9-9ec1-a6e7132a1365-console-serving-cert\") pod \"console-96695bc7-4k645\" (UID: \"d90ced29-8d27-4cd9-9ec1-a6e7132a1365\") " pod="openshift-console/console-96695bc7-4k645" Jul 27 20:35:43.134949 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:43.134932 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf78p\" (UniqueName: \"kubernetes.io/projected/d90ced29-8d27-4cd9-9ec1-a6e7132a1365-kube-api-access-mf78p\") pod \"console-96695bc7-4k645\" (UID: \"d90ced29-8d27-4cd9-9ec1-a6e7132a1365\") " pod="openshift-console/console-96695bc7-4k645" Jul 27 20:35:43.268448 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:43.268424 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-96695bc7-4k645" Jul 27 20:35:43.459128 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:43.459055 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-96695bc7-4k645"] Jul 27 20:35:43.465341 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:35:43.465287 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd90ced29_8d27_4cd9_9ec1_a6e7132a1365.slice/crio-3b56df65dd1cd97973277674e6b864bbc857f72147e22a13e8eb1721c7e4c006 WatchSource:0}: Error finding container 3b56df65dd1cd97973277674e6b864bbc857f72147e22a13e8eb1721c7e4c006: Status 404 returned error can't find the container with id 3b56df65dd1cd97973277674e6b864bbc857f72147e22a13e8eb1721c7e4c006 Jul 27 20:35:44.177909 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:44.177631 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d8b8f8b4-trk4k" event={"ID":"9d5ccffd-8b3b-45cb-a4d5-c856fff6bb12","Type":"ContainerStarted","Data":"42b1564d3fd20a10cab6c39ce904ebc70745da7a23a3a4911aae663a84b79f5c"} Jul 27 20:35:44.179820 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:44.179783 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zrhx5" event={"ID":"c3561079-4fcf-46be-8d0d-9a20c39be6d9","Type":"ContainerStarted","Data":"927acccc60a2bcb37208e5b6f33f9069edcfce1a811f60f0f805f9ce85319edd"} Jul 27 20:35:44.183207 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:44.182758 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-6b59f" event={"ID":"c79f7f0b-c4e0-424f-9067-10f0ffe0963b","Type":"ContainerStarted","Data":"fe9a59a2775a451a2e0794d2215fcbf3b60684172302d54951d05c93a9cc38e1"} Jul 27 20:35:44.183207 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:44.182811 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-6b59f" event={"ID":"c79f7f0b-c4e0-424f-9067-10f0ffe0963b","Type":"ContainerStarted","Data":"1694bf43c4f6b67a44ac88fc1a8d07f286dbd9ba638ad9cef1e345369da53754"} Jul 27 20:35:44.184941 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:44.184875 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-96695bc7-4k645" event={"ID":"d90ced29-8d27-4cd9-9ec1-a6e7132a1365","Type":"ContainerStarted","Data":"3b56df65dd1cd97973277674e6b864bbc857f72147e22a13e8eb1721c7e4c006"} Jul 27 20:35:44.186847 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:44.186813 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j42rs" event={"ID":"58bd29b9-0422-4c07-90f2-04d39cd07db5","Type":"ContainerStarted","Data":"df42b8c51ca0d87ec3af665357feeda80c212992df9c2a74220fb67aa13c0124"} Jul 27 20:35:44.187125 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:44.187105 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-j42rs" Jul 27 20:35:44.193201 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:44.193151 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d8b8f8b4-trk4k" podStartSLOduration=2.243350463 podStartE2EDuration="16.193125478s" totalImagesPullingTime="13.949775015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:35:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:35:44.192877661 +0000 UTC m=+73.928857973" watchObservedRunningTime="2026-07-27 20:35:44.193125478 +0000 UTC m=+73.929105746" Jul 27 20:35:44.210654 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:44.210567 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-6b59f" podStartSLOduration=1.514738818 podStartE2EDuration="6.210546478s" totalImagesPullingTime="4.69580766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:35:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:35:44.209846085 +0000 UTC m=+73.945826395" watchObservedRunningTime="2026-07-27 20:35:44.210546478 +0000 UTC m=+73.946526766" Jul 27 20:35:44.225901 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:44.225839 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-j42rs" podStartSLOduration=66.901335852 podStartE2EDuration="1m13.225825371s" totalImagesPullingTime="6.324489519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:34:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:35:44.225310867 +0000 UTC m=+73.961291154" watchObservedRunningTime="2026-07-27 20:35:44.225825371 +0000 UTC m=+73.961805695" Jul 27 20:35:44.241717 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:44.241666 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zrhx5" podStartSLOduration=66.887455479 podStartE2EDuration="1m13.241650209s" totalImagesPullingTime="6.35419473s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:34:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:35:44.240630777 +0000 UTC m=+73.976611068" watchObservedRunningTime="2026-07-27 20:35:44.241650209 +0000 UTC m=+73.977630495" Jul 27 20:35:46.447645 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.447589 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mk7dq"] Jul 27 20:35:46.476654 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.475118 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-qn6t9"] Jul 27 20:35:46.477990 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.476964 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mk7dq" Jul 27 20:35:46.485633 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.481720 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-f459l\"" Jul 27 20:35:46.485633 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.482004 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 27 20:35:46.485633 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.482223 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 27 20:35:46.485633 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.482390 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 27 20:35:46.492949 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.491796 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-qn6t9"] Jul 27 20:35:46.492949 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.491932 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qn6t9" Jul 27 20:35:46.503706 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.501374 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 27 20:35:46.503706 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.501639 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 27 20:35:46.503706 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.501873 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4ntrv\"" Jul 27 20:35:46.503706 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.502104 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 27 20:35:46.562352 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.562308 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/212e1f86-6abe-442e-a852-3885e1db697e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-qn6t9\" (UID: \"212e1f86-6abe-442e-a852-3885e1db697e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qn6t9" Jul 27 20:35:46.562555 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.562372 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f571cef4-c931-4351-b2af-11d41a5d5d6f-metrics-client-ca\") pod \"node-exporter-mk7dq\" (UID: \"f571cef4-c931-4351-b2af-11d41a5d5d6f\") " pod="openshift-monitoring/node-exporter-mk7dq" Jul 27 20:35:46.562555 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.562400 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f571cef4-c931-4351-b2af-11d41a5d5d6f-node-exporter-accelerators-collector-config\") pod \"node-exporter-mk7dq\" (UID: \"f571cef4-c931-4351-b2af-11d41a5d5d6f\") " pod="openshift-monitoring/node-exporter-mk7dq" Jul 27 20:35:46.562555 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.562448 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/212e1f86-6abe-442e-a852-3885e1db697e-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-qn6t9\" (UID: \"212e1f86-6abe-442e-a852-3885e1db697e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qn6t9" Jul 27 20:35:46.562555 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.562477 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/212e1f86-6abe-442e-a852-3885e1db697e-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-qn6t9\" (UID: \"212e1f86-6abe-442e-a852-3885e1db697e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qn6t9" Jul 27 20:35:46.562555 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.562501 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/212e1f86-6abe-442e-a852-3885e1db697e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-qn6t9\" (UID: \"212e1f86-6abe-442e-a852-3885e1db697e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qn6t9" Jul 27 20:35:46.562555 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.562529 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f571cef4-c931-4351-b2af-11d41a5d5d6f-sys\") pod \"node-exporter-mk7dq\" (UID: \"f571cef4-c931-4351-b2af-11d41a5d5d6f\") " pod="openshift-monitoring/node-exporter-mk7dq" Jul 27 20:35:46.562555 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.562552 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f571cef4-c931-4351-b2af-11d41a5d5d6f-node-exporter-textfile\") pod \"node-exporter-mk7dq\" (UID: \"f571cef4-c931-4351-b2af-11d41a5d5d6f\") " pod="openshift-monitoring/node-exporter-mk7dq" Jul 27 20:35:46.562852 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.562576 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f571cef4-c931-4351-b2af-11d41a5d5d6f-node-exporter-tls\") pod \"node-exporter-mk7dq\" (UID: \"f571cef4-c931-4351-b2af-11d41a5d5d6f\") " pod="openshift-monitoring/node-exporter-mk7dq" Jul 27 20:35:46.562852 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.562618 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/212e1f86-6abe-442e-a852-3885e1db697e-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-qn6t9\" (UID: \"212e1f86-6abe-442e-a852-3885e1db697e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qn6t9" Jul 27 20:35:46.562852 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.562641 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f571cef4-c931-4351-b2af-11d41a5d5d6f-node-exporter-wtmp\") pod \"node-exporter-mk7dq\" (UID: \"f571cef4-c931-4351-b2af-11d41a5d5d6f\") " pod="openshift-monitoring/node-exporter-mk7dq" Jul 27 20:35:46.562852 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.562686 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f571cef4-c931-4351-b2af-11d41a5d5d6f-root\") pod \"node-exporter-mk7dq\" (UID: \"f571cef4-c931-4351-b2af-11d41a5d5d6f\") " pod="openshift-monitoring/node-exporter-mk7dq" Jul 27 20:35:46.562852 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.562716 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxhdx\" (UniqueName: \"kubernetes.io/projected/f571cef4-c931-4351-b2af-11d41a5d5d6f-kube-api-access-zxhdx\") pod \"node-exporter-mk7dq\" (UID: \"f571cef4-c931-4351-b2af-11d41a5d5d6f\") " pod="openshift-monitoring/node-exporter-mk7dq" Jul 27 20:35:46.562852 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.562751 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb85n\" (UniqueName: \"kubernetes.io/projected/212e1f86-6abe-442e-a852-3885e1db697e-kube-api-access-sb85n\") pod \"kube-state-metrics-56d9f944b4-qn6t9\" (UID: \"212e1f86-6abe-442e-a852-3885e1db697e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qn6t9" Jul 27 20:35:46.562852 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.562786 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f571cef4-c931-4351-b2af-11d41a5d5d6f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mk7dq\" (UID: \"f571cef4-c931-4351-b2af-11d41a5d5d6f\") " pod="openshift-monitoring/node-exporter-mk7dq" Jul 27 20:35:46.663547 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.663504 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/212e1f86-6abe-442e-a852-3885e1db697e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-qn6t9\" (UID: \"212e1f86-6abe-442e-a852-3885e1db697e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qn6t9" Jul 27 20:35:46.663755 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.663559 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f571cef4-c931-4351-b2af-11d41a5d5d6f-metrics-client-ca\") pod \"node-exporter-mk7dq\" (UID: \"f571cef4-c931-4351-b2af-11d41a5d5d6f\") " pod="openshift-monitoring/node-exporter-mk7dq" Jul 27 20:35:46.663755 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.663577 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f571cef4-c931-4351-b2af-11d41a5d5d6f-node-exporter-accelerators-collector-config\") pod \"node-exporter-mk7dq\" (UID: \"f571cef4-c931-4351-b2af-11d41a5d5d6f\") " pod="openshift-monitoring/node-exporter-mk7dq" Jul 27 20:35:46.663755 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.663630 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/212e1f86-6abe-442e-a852-3885e1db697e-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-qn6t9\" (UID: \"212e1f86-6abe-442e-a852-3885e1db697e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qn6t9" Jul 27 20:35:46.663755 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.663658 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/212e1f86-6abe-442e-a852-3885e1db697e-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-qn6t9\" (UID: \"212e1f86-6abe-442e-a852-3885e1db697e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qn6t9" Jul 27 20:35:46.663755 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.663683 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/212e1f86-6abe-442e-a852-3885e1db697e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-qn6t9\" (UID: \"212e1f86-6abe-442e-a852-3885e1db697e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qn6t9" Jul 27 20:35:46.663755 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.663709 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f571cef4-c931-4351-b2af-11d41a5d5d6f-sys\") pod \"node-exporter-mk7dq\" (UID: \"f571cef4-c931-4351-b2af-11d41a5d5d6f\") " pod="openshift-monitoring/node-exporter-mk7dq" Jul 27 20:35:46.663755 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.663732 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f571cef4-c931-4351-b2af-11d41a5d5d6f-node-exporter-textfile\") pod \"node-exporter-mk7dq\" (UID: \"f571cef4-c931-4351-b2af-11d41a5d5d6f\") " pod="openshift-monitoring/node-exporter-mk7dq" Jul 27 20:35:46.663755 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.663754 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f571cef4-c931-4351-b2af-11d41a5d5d6f-node-exporter-tls\") pod \"node-exporter-mk7dq\" (UID: \"f571cef4-c931-4351-b2af-11d41a5d5d6f\") " pod="openshift-monitoring/node-exporter-mk7dq" Jul 27 20:35:46.664163 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.663773 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/212e1f86-6abe-442e-a852-3885e1db697e-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-qn6t9\" (UID: \"212e1f86-6abe-442e-a852-3885e1db697e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qn6t9" Jul 27 20:35:46.664163 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.663816 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f571cef4-c931-4351-b2af-11d41a5d5d6f-node-exporter-wtmp\") pod \"node-exporter-mk7dq\" (UID: \"f571cef4-c931-4351-b2af-11d41a5d5d6f\") " pod="openshift-monitoring/node-exporter-mk7dq" Jul 27 20:35:46.664163 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.663873 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f571cef4-c931-4351-b2af-11d41a5d5d6f-root\") pod \"node-exporter-mk7dq\" (UID: \"f571cef4-c931-4351-b2af-11d41a5d5d6f\") " pod="openshift-monitoring/node-exporter-mk7dq" Jul 27 20:35:46.664163 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.663898 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxhdx\" (UniqueName: \"kubernetes.io/projected/f571cef4-c931-4351-b2af-11d41a5d5d6f-kube-api-access-zxhdx\") pod \"node-exporter-mk7dq\" (UID: \"f571cef4-c931-4351-b2af-11d41a5d5d6f\") " pod="openshift-monitoring/node-exporter-mk7dq" Jul 27 20:35:46.664163 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.663943 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb85n\" (UniqueName: \"kubernetes.io/projected/212e1f86-6abe-442e-a852-3885e1db697e-kube-api-access-sb85n\") pod \"kube-state-metrics-56d9f944b4-qn6t9\" (UID: \"212e1f86-6abe-442e-a852-3885e1db697e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qn6t9" Jul 27 20:35:46.664163 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.663986 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f571cef4-c931-4351-b2af-11d41a5d5d6f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mk7dq\" (UID: \"f571cef4-c931-4351-b2af-11d41a5d5d6f\") " pod="openshift-monitoring/node-exporter-mk7dq" Jul 27 20:35:46.664163 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.664031 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/212e1f86-6abe-442e-a852-3885e1db697e-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-qn6t9\" (UID: \"212e1f86-6abe-442e-a852-3885e1db697e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qn6t9" Jul 27 20:35:46.664525 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.664209 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f571cef4-c931-4351-b2af-11d41a5d5d6f-metrics-client-ca\") pod \"node-exporter-mk7dq\" (UID: \"f571cef4-c931-4351-b2af-11d41a5d5d6f\") " pod="openshift-monitoring/node-exporter-mk7dq" Jul 27 20:35:46.664525 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.664279 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f571cef4-c931-4351-b2af-11d41a5d5d6f-root\") pod \"node-exporter-mk7dq\" (UID: \"f571cef4-c931-4351-b2af-11d41a5d5d6f\") " pod="openshift-monitoring/node-exporter-mk7dq" Jul 27 20:35:46.664525 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.664290 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f571cef4-c931-4351-b2af-11d41a5d5d6f-node-exporter-accelerators-collector-config\") pod \"node-exporter-mk7dq\" (UID: \"f571cef4-c931-4351-b2af-11d41a5d5d6f\") " pod="openshift-monitoring/node-exporter-mk7dq" Jul 27 20:35:46.664525 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:46.664349 2570 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 27 20:35:46.664525 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.664349 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/212e1f86-6abe-442e-a852-3885e1db697e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-qn6t9\" (UID: \"212e1f86-6abe-442e-a852-3885e1db697e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qn6t9" Jul 27 20:35:46.664525 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:46.664404 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/212e1f86-6abe-442e-a852-3885e1db697e-kube-state-metrics-tls podName:212e1f86-6abe-442e-a852-3885e1db697e nodeName:}" failed. No retries permitted until 2026-07-27 20:35:47.164386638 +0000 UTC m=+76.900366908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/212e1f86-6abe-442e-a852-3885e1db697e-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-qn6t9" (UID: "212e1f86-6abe-442e-a852-3885e1db697e") : secret "kube-state-metrics-tls" not found Jul 27 20:35:46.664525 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.664440 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f571cef4-c931-4351-b2af-11d41a5d5d6f-node-exporter-wtmp\") pod \"node-exporter-mk7dq\" (UID: \"f571cef4-c931-4351-b2af-11d41a5d5d6f\") " pod="openshift-monitoring/node-exporter-mk7dq" Jul 27 20:35:46.664896 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.664570 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f571cef4-c931-4351-b2af-11d41a5d5d6f-sys\") pod \"node-exporter-mk7dq\" (UID: \"f571cef4-c931-4351-b2af-11d41a5d5d6f\") " pod="openshift-monitoring/node-exporter-mk7dq" Jul 27 20:35:46.664896 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.664845 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f571cef4-c931-4351-b2af-11d41a5d5d6f-node-exporter-textfile\") pod \"node-exporter-mk7dq\" (UID: \"f571cef4-c931-4351-b2af-11d41a5d5d6f\") " pod="openshift-monitoring/node-exporter-mk7dq" Jul 27 20:35:46.665293 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.665251 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/212e1f86-6abe-442e-a852-3885e1db697e-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-qn6t9\" (UID: \"212e1f86-6abe-442e-a852-3885e1db697e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qn6t9" Jul 27 20:35:46.667212 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.667189 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f571cef4-c931-4351-b2af-11d41a5d5d6f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mk7dq\" (UID: \"f571cef4-c931-4351-b2af-11d41a5d5d6f\") " pod="openshift-monitoring/node-exporter-mk7dq" Jul 27 20:35:46.667301 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.667189 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/212e1f86-6abe-442e-a852-3885e1db697e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-qn6t9\" (UID: \"212e1f86-6abe-442e-a852-3885e1db697e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qn6t9" Jul 27 20:35:46.667358 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.667333 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f571cef4-c931-4351-b2af-11d41a5d5d6f-node-exporter-tls\") pod \"node-exporter-mk7dq\" (UID: \"f571cef4-c931-4351-b2af-11d41a5d5d6f\") " pod="openshift-monitoring/node-exporter-mk7dq" Jul 27 20:35:46.676853 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.676831 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxhdx\" (UniqueName: \"kubernetes.io/projected/f571cef4-c931-4351-b2af-11d41a5d5d6f-kube-api-access-zxhdx\") pod \"node-exporter-mk7dq\" (UID: \"f571cef4-c931-4351-b2af-11d41a5d5d6f\") " pod="openshift-monitoring/node-exporter-mk7dq" Jul 27 20:35:46.678004 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.677981 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb85n\" (UniqueName: \"kubernetes.io/projected/212e1f86-6abe-442e-a852-3885e1db697e-kube-api-access-sb85n\") pod \"kube-state-metrics-56d9f944b4-qn6t9\" (UID: \"212e1f86-6abe-442e-a852-3885e1db697e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qn6t9" Jul 27 20:35:46.794743 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:46.794714 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mk7dq" Jul 27 20:35:46.805357 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:35:46.805316 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf571cef4_c931_4351_b2af_11d41a5d5d6f.slice/crio-36dacba2f00e1ebd6678d236936295c7e7a76de765ad696abeefa6fba535c23d WatchSource:0}: Error finding container 36dacba2f00e1ebd6678d236936295c7e7a76de765ad696abeefa6fba535c23d: Status 404 returned error can't find the container with id 36dacba2f00e1ebd6678d236936295c7e7a76de765ad696abeefa6fba535c23d Jul 27 20:35:47.167733 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.167681 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/212e1f86-6abe-442e-a852-3885e1db697e-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-qn6t9\" (UID: \"212e1f86-6abe-442e-a852-3885e1db697e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qn6t9" Jul 27 20:35:47.170296 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.170270 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/212e1f86-6abe-442e-a852-3885e1db697e-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-qn6t9\" (UID: \"212e1f86-6abe-442e-a852-3885e1db697e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qn6t9" Jul 27 20:35:47.205460 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.205351 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-96695bc7-4k645" event={"ID":"d90ced29-8d27-4cd9-9ec1-a6e7132a1365","Type":"ContainerStarted","Data":"493271fc696fa2dd2172685aebd7aa0023bd1cfcd35099834a6ef4bfa67e3da1"} Jul 27 20:35:47.207329 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.207289 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mk7dq" event={"ID":"f571cef4-c931-4351-b2af-11d41a5d5d6f","Type":"ContainerStarted","Data":"36dacba2f00e1ebd6678d236936295c7e7a76de765ad696abeefa6fba535c23d"} Jul 27 20:35:47.224813 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.224761 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-96695bc7-4k645" podStartSLOduration=1.990353426 podStartE2EDuration="5.224742118s" totalImagesPullingTime="3.234388692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:35:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:35:47.224429641 +0000 UTC m=+76.960409929" watchObservedRunningTime="2026-07-27 20:35:47.224742118 +0000 UTC m=+76.960722398" Jul 27 20:35:47.408225 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.408188 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qn6t9" Jul 27 20:35:47.495512 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.493428 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 20:35:47.504626 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.499965 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.509586 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.509410 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 27 20:35:47.509586 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.509425 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-dc8tf\"" Jul 27 20:35:47.509842 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.509707 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 27 20:35:47.509842 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.509807 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 27 20:35:47.509941 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.509712 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 27 20:35:47.510580 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.510035 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 27 20:35:47.510580 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.510058 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 27 20:35:47.510580 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.510256 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 27 20:35:47.510580 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.510313 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 27 20:35:47.510580 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.510446 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 27 20:35:47.518326 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.515707 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 20:35:47.578569 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.578537 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.578569 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.578575 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0cedefa7-efaf-490b-b9b8-e3fb34570367-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.578813 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.578632 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.578813 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.578659 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cedefa7-efaf-490b-b9b8-e3fb34570367-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.578813 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.578742 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-config-volume\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.578813 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.578796 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0cedefa7-efaf-490b-b9b8-e3fb34570367-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.578952 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.578831 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0cedefa7-efaf-490b-b9b8-e3fb34570367-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.578952 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.578876 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-web-config\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.578952 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.578941 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.579104 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.578983 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.579104 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.579020 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0cedefa7-efaf-490b-b9b8-e3fb34570367-config-out\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.579104 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.579051 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.579212 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.579115 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fd7n\" (UniqueName: \"kubernetes.io/projected/0cedefa7-efaf-490b-b9b8-e3fb34570367-kube-api-access-9fd7n\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.590548 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.589113 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-qn6t9"] Jul 27 20:35:47.679987 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.679915 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-web-config\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.680177 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.679986 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.680177 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.680020 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.680177 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.680047 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0cedefa7-efaf-490b-b9b8-e3fb34570367-config-out\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.680177 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.680069 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.680177 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.680119 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fd7n\" (UniqueName: \"kubernetes.io/projected/0cedefa7-efaf-490b-b9b8-e3fb34570367-kube-api-access-9fd7n\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.680177 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.680172 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.680477 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.680197 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0cedefa7-efaf-490b-b9b8-e3fb34570367-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.680477 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.680228 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.680477 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.680252 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cedefa7-efaf-490b-b9b8-e3fb34570367-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.680477 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.680279 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-config-volume\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.680477 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.680305 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0cedefa7-efaf-490b-b9b8-e3fb34570367-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.680477 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.680331 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0cedefa7-efaf-490b-b9b8-e3fb34570367-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.681168 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.681124 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0cedefa7-efaf-490b-b9b8-e3fb34570367-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.682223 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.682174 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0cedefa7-efaf-490b-b9b8-e3fb34570367-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.684286 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.684258 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cedefa7-efaf-490b-b9b8-e3fb34570367-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.685850 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.685825 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0cedefa7-efaf-490b-b9b8-e3fb34570367-config-out\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.686527 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.686504 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-web-config\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.686850 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.686827 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.687735 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.687706 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.687837 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.687760 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.688403 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.688379 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0cedefa7-efaf-490b-b9b8-e3fb34570367-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.688492 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.688472 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.688593 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.688570 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.689204 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.689162 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-config-volume\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.690561 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.690540 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fd7n\" (UniqueName: \"kubernetes.io/projected/0cedefa7-efaf-490b-b9b8-e3fb34570367-kube-api-access-9fd7n\") pod \"alertmanager-main-0\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.804682 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:35:47.804641 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod212e1f86_6abe_442e_a852_3885e1db697e.slice/crio-df8a2f8e129e83f30b5ebbb67cf3faa33cc260d807cecdb1bdc3c7cdd7e0af29 WatchSource:0}: Error finding container df8a2f8e129e83f30b5ebbb67cf3faa33cc260d807cecdb1bdc3c7cdd7e0af29: Status 404 returned error can't find the container with id df8a2f8e129e83f30b5ebbb67cf3faa33cc260d807cecdb1bdc3c7cdd7e0af29 Jul 27 20:35:47.821795 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.821765 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:35:47.881969 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.881933 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4xkz\" (UID: \"a4512db5-a1ae-4484-be6e-cd5b7b3046ee\") " pod="openshift-insights/insights-runtime-extractor-s4xkz" Jul 27 20:35:47.881969 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.881986 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4xkz\" (UID: \"a4512db5-a1ae-4484-be6e-cd5b7b3046ee\") " pod="openshift-insights/insights-runtime-extractor-s4xkz" Jul 27 20:35:47.882452 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.882431 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4xkz\" (UID: \"a4512db5-a1ae-4484-be6e-cd5b7b3046ee\") " pod="openshift-insights/insights-runtime-extractor-s4xkz" Jul 27 20:35:47.884399 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.884373 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a4512db5-a1ae-4484-be6e-cd5b7b3046ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4xkz\" (UID: \"a4512db5-a1ae-4484-be6e-cd5b7b3046ee\") " pod="openshift-insights/insights-runtime-extractor-s4xkz" Jul 27 20:35:47.993910 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:47.993704 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 20:35:47.996638 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:35:47.996589 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cedefa7_efaf_490b_b9b8_e3fb34570367.slice/crio-17d7e2a30ec33067cdbf89ab2c2e0ebb3c000edcbfb2fdd0a700c39082345dc5 WatchSource:0}: Error finding container 17d7e2a30ec33067cdbf89ab2c2e0ebb3c000edcbfb2fdd0a700c39082345dc5: Status 404 returned error can't find the container with id 17d7e2a30ec33067cdbf89ab2c2e0ebb3c000edcbfb2fdd0a700c39082345dc5 Jul 27 20:35:48.026079 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:48.026052 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s4xkz" Jul 27 20:35:48.160986 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:48.160960 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s4xkz"] Jul 27 20:35:48.164235 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:35:48.164211 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4512db5_a1ae_4484_be6e_cd5b7b3046ee.slice/crio-96e2b6971e8678d38ca7083b6eec92f7a8a8464bafe6a02940869907e993dc47 WatchSource:0}: Error finding container 96e2b6971e8678d38ca7083b6eec92f7a8a8464bafe6a02940869907e993dc47: Status 404 returned error can't find the container with id 96e2b6971e8678d38ca7083b6eec92f7a8a8464bafe6a02940869907e993dc47 Jul 27 20:35:48.213105 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:48.213072 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0cedefa7-efaf-490b-b9b8-e3fb34570367","Type":"ContainerStarted","Data":"17d7e2a30ec33067cdbf89ab2c2e0ebb3c000edcbfb2fdd0a700c39082345dc5"} Jul 27 20:35:48.215644 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:48.215328 2570 generic.go:356] "Generic (PLEG): container finished" podID="f571cef4-c931-4351-b2af-11d41a5d5d6f" containerID="22a14869412ccfcb7151144b19ccf6aa05cddceda4f41a61e90d732cd992284d" exitCode=0 Jul 27 20:35:48.215644 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:48.215378 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mk7dq" event={"ID":"f571cef4-c931-4351-b2af-11d41a5d5d6f","Type":"ContainerDied","Data":"22a14869412ccfcb7151144b19ccf6aa05cddceda4f41a61e90d732cd992284d"} Jul 27 20:35:48.217231 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:48.217199 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s4xkz" event={"ID":"a4512db5-a1ae-4484-be6e-cd5b7b3046ee","Type":"ContainerStarted","Data":"96e2b6971e8678d38ca7083b6eec92f7a8a8464bafe6a02940869907e993dc47"} Jul 27 20:35:48.218759 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:48.218732 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qn6t9" event={"ID":"212e1f86-6abe-442e-a852-3885e1db697e","Type":"ContainerStarted","Data":"df8a2f8e129e83f30b5ebbb67cf3faa33cc260d807cecdb1bdc3c7cdd7e0af29"} Jul 27 20:35:49.230631 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:49.230357 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mk7dq" event={"ID":"f571cef4-c931-4351-b2af-11d41a5d5d6f","Type":"ContainerStarted","Data":"216f303a8ee0b69fe73539ff5de62ebef46effef7cdeb20714c3a6743f0e8347"} Jul 27 20:35:49.230631 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:49.230420 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mk7dq" event={"ID":"f571cef4-c931-4351-b2af-11d41a5d5d6f","Type":"ContainerStarted","Data":"b44e51cb841f2d1eca61fd7175094498de0864dea4c7c5e0c1d93d26103f3154"} Jul 27 20:35:49.251383 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:49.251319 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mk7dq" podStartSLOduration=2.123889983 podStartE2EDuration="3.251288478s" totalImagesPullingTime="1.044595611s" totalInitContainerRuntime="82.802884ms" isStatefulPod=true podCreationTimestamp="2026-07-27 20:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:35:49.250937116 +0000 UTC m=+78.986917409" watchObservedRunningTime="2026-07-27 20:35:49.251288478 +0000 UTC m=+78.987268746" Jul 27 20:35:50.237542 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:50.237479 2570 generic.go:356] "Generic (PLEG): container finished" podID="0cedefa7-efaf-490b-b9b8-e3fb34570367" containerID="7a5920ea941e129697a6a269970ed9ab85bebf79f858f958376bae983c668e27" exitCode=0 Jul 27 20:35:50.237954 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:50.237671 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0cedefa7-efaf-490b-b9b8-e3fb34570367","Type":"ContainerDied","Data":"7a5920ea941e129697a6a269970ed9ab85bebf79f858f958376bae983c668e27"} Jul 27 20:35:50.240031 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:50.240001 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qn6t9" event={"ID":"212e1f86-6abe-442e-a852-3885e1db697e","Type":"ContainerStarted","Data":"f9ed5dd376a3974ea3d06cec2a4c02d50804b87b18135941de8c821e010a963a"} Jul 27 20:35:51.211772 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:51.211734 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-k2f67"] Jul 27 20:35:51.220975 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:51.220943 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-k2f67" Jul 27 20:35:51.223814 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:51.223463 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-k2f67"] Jul 27 20:35:51.223814 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:51.223515 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 27 20:35:51.223814 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:51.223637 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-6bj8j\"" Jul 27 20:35:51.247990 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:51.247947 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qn6t9" event={"ID":"212e1f86-6abe-442e-a852-3885e1db697e","Type":"ContainerStarted","Data":"4994ebde68df314db48c2806f50b01cdda5dc8eea060563062cdfa2272aefc9f"} Jul 27 20:35:51.248464 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:51.248016 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qn6t9" event={"ID":"212e1f86-6abe-442e-a852-3885e1db697e","Type":"ContainerStarted","Data":"e6c5e582a0f0a104081873916c12b1a2caae4c1b007a79287fef3088a0fbe677"} Jul 27 20:35:51.266923 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:51.266844 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qn6t9" podStartSLOduration=2.958293036 podStartE2EDuration="5.266820338s" totalImagesPullingTime="2.308527302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:35:51.26657386 +0000 UTC m=+81.002554148" watchObservedRunningTime="2026-07-27 20:35:51.266820338 +0000 UTC m=+81.002800626" Jul 27 20:35:51.313817 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:51.313776 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqxdm\" (UniqueName: \"kubernetes.io/projected/4e2dc09e-afb4-433d-83ca-a2add0b7cc2e-kube-api-access-pqxdm\") pod \"monitoring-plugin-68b7fc9574-k2f67\" (UID: \"4e2dc09e-afb4-433d-83ca-a2add0b7cc2e\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-k2f67" Jul 27 20:35:51.314010 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:51.313926 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4e2dc09e-afb4-433d-83ca-a2add0b7cc2e-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-k2f67\" (UID: \"4e2dc09e-afb4-433d-83ca-a2add0b7cc2e\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-k2f67" Jul 27 20:35:51.415113 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:51.415014 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4e2dc09e-afb4-433d-83ca-a2add0b7cc2e-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-k2f67\" (UID: \"4e2dc09e-afb4-433d-83ca-a2add0b7cc2e\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-k2f67" Jul 27 20:35:51.415113 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:51.415109 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqxdm\" (UniqueName: \"kubernetes.io/projected/4e2dc09e-afb4-433d-83ca-a2add0b7cc2e-kube-api-access-pqxdm\") pod \"monitoring-plugin-68b7fc9574-k2f67\" (UID: \"4e2dc09e-afb4-433d-83ca-a2add0b7cc2e\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-k2f67" Jul 27 20:35:51.415340 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:51.415188 2570 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 27 20:35:51.415340 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:35:51.415270 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e2dc09e-afb4-433d-83ca-a2add0b7cc2e-monitoring-plugin-cert podName:4e2dc09e-afb4-433d-83ca-a2add0b7cc2e nodeName:}" failed. No retries permitted until 2026-07-27 20:35:51.915247614 +0000 UTC m=+81.651227892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/4e2dc09e-afb4-433d-83ca-a2add0b7cc2e-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-k2f67" (UID: "4e2dc09e-afb4-433d-83ca-a2add0b7cc2e") : secret "monitoring-plugin-cert" not found Jul 27 20:35:51.425720 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:51.425689 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqxdm\" (UniqueName: \"kubernetes.io/projected/4e2dc09e-afb4-433d-83ca-a2add0b7cc2e-kube-api-access-pqxdm\") pod \"monitoring-plugin-68b7fc9574-k2f67\" (UID: \"4e2dc09e-afb4-433d-83ca-a2add0b7cc2e\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-k2f67" Jul 27 20:35:51.928776 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:51.928623 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4e2dc09e-afb4-433d-83ca-a2add0b7cc2e-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-k2f67\" (UID: \"4e2dc09e-afb4-433d-83ca-a2add0b7cc2e\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-k2f67" Jul 27 20:35:51.931727 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:51.931662 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4e2dc09e-afb4-433d-83ca-a2add0b7cc2e-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-k2f67\" (UID: \"4e2dc09e-afb4-433d-83ca-a2add0b7cc2e\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-k2f67" Jul 27 20:35:52.135813 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:52.135762 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-k2f67" Jul 27 20:35:53.269365 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:53.268565 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-96695bc7-4k645" Jul 27 20:35:53.272074 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:53.270544 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-96695bc7-4k645" Jul 27 20:35:53.278193 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:53.278170 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-96695bc7-4k645" Jul 27 20:35:54.272123 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:54.271867 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-96695bc7-4k645" Jul 27 20:35:54.296859 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:54.296665 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-k2f67"] Jul 27 20:35:55.281190 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:55.280918 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0cedefa7-efaf-490b-b9b8-e3fb34570367","Type":"ContainerStarted","Data":"2e08a7fe270f49c4181868ab8af97cfe14235bcc7298ee83c443845ec7986e86"} Jul 27 20:35:55.281190 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:55.281016 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0cedefa7-efaf-490b-b9b8-e3fb34570367","Type":"ContainerStarted","Data":"256392deb3ad2d9faeb83790f4f41b68f40e834bd841c6458695295b029d430b"} Jul 27 20:35:55.281190 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:55.281060 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0cedefa7-efaf-490b-b9b8-e3fb34570367","Type":"ContainerStarted","Data":"46c79ce1d1071a51103771c8b5c6ba15fdf5603efa41f243442ed930bb83069c"} Jul 27 20:35:55.281190 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:55.281097 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0cedefa7-efaf-490b-b9b8-e3fb34570367","Type":"ContainerStarted","Data":"2065d07b04d958742261ba63f2a6564b91443f9a18330fdd59b7eea6ca0ad5f2"} Jul 27 20:35:55.281190 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:55.281137 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0cedefa7-efaf-490b-b9b8-e3fb34570367","Type":"ContainerStarted","Data":"8358940e2b1c377288038a3be9f75f0b3d9d9693b5835afd66ba90a8d456da97"} Jul 27 20:35:55.282742 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:55.282713 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s4xkz" event={"ID":"a4512db5-a1ae-4484-be6e-cd5b7b3046ee","Type":"ContainerStarted","Data":"e727879bcab581d5b8be9facf15ca8d2ff56aa6c5f4350cd2d6d675c44ccdcfc"} Jul 27 20:35:55.284644 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:55.284618 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-k2f67" event={"ID":"4e2dc09e-afb4-433d-83ca-a2add0b7cc2e","Type":"ContainerStarted","Data":"598fc074be03208f1b1a94fa3555374058d97a64ae3922a80176c2eb8f430cc8"} Jul 27 20:35:58.300454 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:58.300410 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s4xkz" event={"ID":"a4512db5-a1ae-4484-be6e-cd5b7b3046ee","Type":"ContainerStarted","Data":"49ff992a098f2f1bc01eb8aa49d694df7cb24761b21eb90a9d63f763a8f9fd49"} Jul 27 20:35:58.301912 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:58.301886 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-k2f67" event={"ID":"4e2dc09e-afb4-433d-83ca-a2add0b7cc2e","Type":"ContainerStarted","Data":"abb3231e861f704926c6ce5a6d1a8cc487a49819e2796a938b8fe9362555eb2f"} Jul 27 20:35:58.302118 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:58.302094 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-k2f67" Jul 27 20:35:58.305845 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:58.305823 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0cedefa7-efaf-490b-b9b8-e3fb34570367","Type":"ContainerStarted","Data":"80b02820c600b83907d7537b98b951b6e1b2f550e823b8d4298c4af37e0ecaea"} Jul 27 20:35:58.307903 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:58.307884 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-k2f67" Jul 27 20:35:58.317329 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:58.317280 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-k2f67" podStartSLOduration=3.748225046 podStartE2EDuration="7.317265231s" totalImagesPullingTime="3.569040185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:35:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:35:58.316094111 +0000 UTC m=+88.052074398" watchObservedRunningTime="2026-07-27 20:35:58.317265231 +0000 UTC m=+88.053245496" Jul 27 20:35:58.356369 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:35:58.356313 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.978447603 podStartE2EDuration="11.356290028s" totalImagesPullingTime="9.354735512s" totalInitContainerRuntime="23.106913ms" isStatefulPod=false podCreationTimestamp="2026-07-27 20:35:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:35:58.354281958 +0000 UTC m=+88.090262245" watchObservedRunningTime="2026-07-27 20:35:58.356290028 +0000 UTC m=+88.092270315" Jul 27 20:36:02.326299 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:36:02.326263 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s4xkz" event={"ID":"a4512db5-a1ae-4484-be6e-cd5b7b3046ee","Type":"ContainerStarted","Data":"077f984bd2b1299556190c522c6e208fb96293c95290e8977886da26b219551f"} Jul 27 20:36:02.347028 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:36:02.346968 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-s4xkz" podStartSLOduration=17.013545796 podStartE2EDuration="30.346949929s" totalImagesPullingTime="13.333404133s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:35:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:36:02.345215629 +0000 UTC m=+92.081195910" watchObservedRunningTime="2026-07-27 20:36:02.346949929 +0000 UTC m=+92.082930248" Jul 27 20:36:12.463501 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:36:12.463463 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-96695bc7-4k645"] Jul 27 20:36:15.194005 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:36:15.193974 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-j42rs" Jul 27 20:36:37.484521 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:36:37.484464 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-96695bc7-4k645" podUID="d90ced29-8d27-4cd9-9ec1-a6e7132a1365" containerName="console" containerID="cri-o://493271fc696fa2dd2172685aebd7aa0023bd1cfcd35099834a6ef4bfa67e3da1" gracePeriod=15 Jul 27 20:36:37.720298 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:36:37.720276 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-96695bc7-4k645" Jul 27 20:36:37.790623 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:36:37.790506 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mf78p\" (UniqueName: \"kubernetes.io/projected/d90ced29-8d27-4cd9-9ec1-a6e7132a1365-kube-api-access-mf78p\") pod \"d90ced29-8d27-4cd9-9ec1-a6e7132a1365\" (UID: \"d90ced29-8d27-4cd9-9ec1-a6e7132a1365\") " Jul 27 20:36:37.790623 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:36:37.790547 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d90ced29-8d27-4cd9-9ec1-a6e7132a1365-console-serving-cert\") pod \"d90ced29-8d27-4cd9-9ec1-a6e7132a1365\" (UID: \"d90ced29-8d27-4cd9-9ec1-a6e7132a1365\") " Jul 27 20:36:37.790835 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:36:37.790629 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d90ced29-8d27-4cd9-9ec1-a6e7132a1365-console-config\") pod \"d90ced29-8d27-4cd9-9ec1-a6e7132a1365\" (UID: \"d90ced29-8d27-4cd9-9ec1-a6e7132a1365\") " Jul 27 20:36:37.790835 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:36:37.790662 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d90ced29-8d27-4cd9-9ec1-a6e7132a1365-service-ca\") pod \"d90ced29-8d27-4cd9-9ec1-a6e7132a1365\" (UID: \"d90ced29-8d27-4cd9-9ec1-a6e7132a1365\") " Jul 27 20:36:37.790835 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:36:37.790692 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d90ced29-8d27-4cd9-9ec1-a6e7132a1365-oauth-serving-cert\") pod \"d90ced29-8d27-4cd9-9ec1-a6e7132a1365\" (UID: \"d90ced29-8d27-4cd9-9ec1-a6e7132a1365\") " Jul 27 20:36:37.790976 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:36:37.790876 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d90ced29-8d27-4cd9-9ec1-a6e7132a1365-console-oauth-config\") pod \"d90ced29-8d27-4cd9-9ec1-a6e7132a1365\" (UID: \"d90ced29-8d27-4cd9-9ec1-a6e7132a1365\") " Jul 27 20:36:37.791090 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:36:37.791063 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d90ced29-8d27-4cd9-9ec1-a6e7132a1365-console-config" (OuterVolumeSpecName: "console-config") pod "d90ced29-8d27-4cd9-9ec1-a6e7132a1365" (UID: "d90ced29-8d27-4cd9-9ec1-a6e7132a1365"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:36:37.791152 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:36:37.791085 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d90ced29-8d27-4cd9-9ec1-a6e7132a1365-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d90ced29-8d27-4cd9-9ec1-a6e7132a1365" (UID: "d90ced29-8d27-4cd9-9ec1-a6e7132a1365"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:36:37.791152 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:36:37.791089 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d90ced29-8d27-4cd9-9ec1-a6e7132a1365-service-ca" (OuterVolumeSpecName: "service-ca") pod "d90ced29-8d27-4cd9-9ec1-a6e7132a1365" (UID: "d90ced29-8d27-4cd9-9ec1-a6e7132a1365"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:36:37.791250 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:36:37.791186 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d90ced29-8d27-4cd9-9ec1-a6e7132a1365-console-config\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:36:37.791250 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:36:37.791208 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d90ced29-8d27-4cd9-9ec1-a6e7132a1365-service-ca\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:36:37.791250 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:36:37.791227 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d90ced29-8d27-4cd9-9ec1-a6e7132a1365-oauth-serving-cert\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:36:37.793195 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:36:37.793161 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d90ced29-8d27-4cd9-9ec1-a6e7132a1365-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d90ced29-8d27-4cd9-9ec1-a6e7132a1365" (UID: "d90ced29-8d27-4cd9-9ec1-a6e7132a1365"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 20:36:37.793314 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:36:37.793197 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d90ced29-8d27-4cd9-9ec1-a6e7132a1365-kube-api-access-mf78p" (OuterVolumeSpecName: "kube-api-access-mf78p") pod "d90ced29-8d27-4cd9-9ec1-a6e7132a1365" (UID: "d90ced29-8d27-4cd9-9ec1-a6e7132a1365"). InnerVolumeSpecName "kube-api-access-mf78p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:36:37.793314 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:36:37.793209 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d90ced29-8d27-4cd9-9ec1-a6e7132a1365-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d90ced29-8d27-4cd9-9ec1-a6e7132a1365" (UID: "d90ced29-8d27-4cd9-9ec1-a6e7132a1365"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 20:36:37.891763 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:36:37.891729 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf78p\" (UniqueName: \"kubernetes.io/projected/d90ced29-8d27-4cd9-9ec1-a6e7132a1365-kube-api-access-mf78p\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:36:37.891763 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:36:37.891758 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d90ced29-8d27-4cd9-9ec1-a6e7132a1365-console-serving-cert\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:36:37.891763 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:36:37.891767 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d90ced29-8d27-4cd9-9ec1-a6e7132a1365-console-oauth-config\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:36:38.468092 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:36:38.468052 2570 generic.go:356] "Generic (PLEG): container finished" podID="d90ced29-8d27-4cd9-9ec1-a6e7132a1365" containerID="493271fc696fa2dd2172685aebd7aa0023bd1cfcd35099834a6ef4bfa67e3da1" exitCode=0 Jul 27 20:36:38.468295 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:36:38.468100 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-96695bc7-4k645" Jul 27 20:36:38.468295 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:36:38.468129 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-96695bc7-4k645" event={"ID":"d90ced29-8d27-4cd9-9ec1-a6e7132a1365","Type":"ContainerDied","Data":"493271fc696fa2dd2172685aebd7aa0023bd1cfcd35099834a6ef4bfa67e3da1"} Jul 27 20:36:38.468295 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:36:38.468181 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-96695bc7-4k645" event={"ID":"d90ced29-8d27-4cd9-9ec1-a6e7132a1365","Type":"ContainerDied","Data":"3b56df65dd1cd97973277674e6b864bbc857f72147e22a13e8eb1721c7e4c006"} Jul 27 20:36:38.468295 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:36:38.468215 2570 scope.go:117] "RemoveContainer" containerID="493271fc696fa2dd2172685aebd7aa0023bd1cfcd35099834a6ef4bfa67e3da1" Jul 27 20:36:38.477295 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:36:38.477275 2570 scope.go:117] "RemoveContainer" containerID="493271fc696fa2dd2172685aebd7aa0023bd1cfcd35099834a6ef4bfa67e3da1" Jul 27 20:36:38.477564 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:36:38.477545 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"493271fc696fa2dd2172685aebd7aa0023bd1cfcd35099834a6ef4bfa67e3da1\": container with ID starting with 493271fc696fa2dd2172685aebd7aa0023bd1cfcd35099834a6ef4bfa67e3da1 not found: ID does not exist" containerID="493271fc696fa2dd2172685aebd7aa0023bd1cfcd35099834a6ef4bfa67e3da1" Jul 27 20:36:38.477634 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:36:38.477576 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"493271fc696fa2dd2172685aebd7aa0023bd1cfcd35099834a6ef4bfa67e3da1"} err="failed to get container status \"493271fc696fa2dd2172685aebd7aa0023bd1cfcd35099834a6ef4bfa67e3da1\": rpc error: code = NotFound desc = could not find container \"493271fc696fa2dd2172685aebd7aa0023bd1cfcd35099834a6ef4bfa67e3da1\": container with ID starting with 493271fc696fa2dd2172685aebd7aa0023bd1cfcd35099834a6ef4bfa67e3da1 not found: ID does not exist" Jul 27 20:36:38.488667 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:36:38.488639 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-96695bc7-4k645"] Jul 27 20:36:38.492245 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:36:38.492222 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-96695bc7-4k645"] Jul 27 20:36:38.852348 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:36:38.852271 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d90ced29-8d27-4cd9-9ec1-a6e7132a1365" path="/var/lib/kubelet/pods/d90ced29-8d27-4cd9-9ec1-a6e7132a1365/volumes" Jul 27 20:37:06.708163 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:06.708127 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 20:37:06.708739 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:06.708442 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0cedefa7-efaf-490b-b9b8-e3fb34570367" containerName="alertmanager" containerID="cri-o://2e08a7fe270f49c4181868ab8af97cfe14235bcc7298ee83c443845ec7986e86" gracePeriod=120 Jul 27 20:37:06.708739 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:06.708577 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0cedefa7-efaf-490b-b9b8-e3fb34570367" containerName="kube-rbac-proxy" containerID="cri-o://46c79ce1d1071a51103771c8b5c6ba15fdf5603efa41f243442ed930bb83069c" gracePeriod=120 Jul 27 20:37:06.708739 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:06.708628 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0cedefa7-efaf-490b-b9b8-e3fb34570367" containerName="prom-label-proxy" containerID="cri-o://80b02820c600b83907d7537b98b951b6e1b2f550e823b8d4298c4af37e0ecaea" gracePeriod=120 Jul 27 20:37:06.708936 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:06.708507 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0cedefa7-efaf-490b-b9b8-e3fb34570367" containerName="kube-rbac-proxy-metric" containerID="cri-o://256392deb3ad2d9faeb83790f4f41b68f40e834bd841c6458695295b029d430b" gracePeriod=120 Jul 27 20:37:06.708936 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:06.708791 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0cedefa7-efaf-490b-b9b8-e3fb34570367" containerName="kube-rbac-proxy-web" containerID="cri-o://2065d07b04d958742261ba63f2a6564b91443f9a18330fdd59b7eea6ca0ad5f2" gracePeriod=120 Jul 27 20:37:06.708936 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:06.708775 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0cedefa7-efaf-490b-b9b8-e3fb34570367" containerName="config-reloader" containerID="cri-o://8358940e2b1c377288038a3be9f75f0b3d9d9693b5835afd66ba90a8d456da97" gracePeriod=120 Jul 27 20:37:07.587521 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:07.587488 2570 generic.go:356] "Generic (PLEG): container finished" podID="0cedefa7-efaf-490b-b9b8-e3fb34570367" containerID="80b02820c600b83907d7537b98b951b6e1b2f550e823b8d4298c4af37e0ecaea" exitCode=0 Jul 27 20:37:07.587521 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:07.587514 2570 generic.go:356] "Generic (PLEG): container finished" podID="0cedefa7-efaf-490b-b9b8-e3fb34570367" containerID="256392deb3ad2d9faeb83790f4f41b68f40e834bd841c6458695295b029d430b" exitCode=0 Jul 27 20:37:07.587521 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:07.587520 2570 generic.go:356] "Generic (PLEG): container finished" podID="0cedefa7-efaf-490b-b9b8-e3fb34570367" containerID="46c79ce1d1071a51103771c8b5c6ba15fdf5603efa41f243442ed930bb83069c" exitCode=0 Jul 27 20:37:07.587521 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:07.587525 2570 generic.go:356] "Generic (PLEG): container finished" podID="0cedefa7-efaf-490b-b9b8-e3fb34570367" containerID="8358940e2b1c377288038a3be9f75f0b3d9d9693b5835afd66ba90a8d456da97" exitCode=0 Jul 27 20:37:07.587521 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:07.587531 2570 generic.go:356] "Generic (PLEG): container finished" podID="0cedefa7-efaf-490b-b9b8-e3fb34570367" containerID="2e08a7fe270f49c4181868ab8af97cfe14235bcc7298ee83c443845ec7986e86" exitCode=0 Jul 27 20:37:07.587860 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:07.587559 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0cedefa7-efaf-490b-b9b8-e3fb34570367","Type":"ContainerDied","Data":"80b02820c600b83907d7537b98b951b6e1b2f550e823b8d4298c4af37e0ecaea"} Jul 27 20:37:07.587860 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:07.587651 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0cedefa7-efaf-490b-b9b8-e3fb34570367","Type":"ContainerDied","Data":"256392deb3ad2d9faeb83790f4f41b68f40e834bd841c6458695295b029d430b"} Jul 27 20:37:07.587860 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:07.587680 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0cedefa7-efaf-490b-b9b8-e3fb34570367","Type":"ContainerDied","Data":"46c79ce1d1071a51103771c8b5c6ba15fdf5603efa41f243442ed930bb83069c"} Jul 27 20:37:07.587860 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:07.587703 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0cedefa7-efaf-490b-b9b8-e3fb34570367","Type":"ContainerDied","Data":"8358940e2b1c377288038a3be9f75f0b3d9d9693b5835afd66ba90a8d456da97"} Jul 27 20:37:07.587860 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:07.587726 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0cedefa7-efaf-490b-b9b8-e3fb34570367","Type":"ContainerDied","Data":"2e08a7fe270f49c4181868ab8af97cfe14235bcc7298ee83c443845ec7986e86"} Jul 27 20:37:07.947967 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:07.947940 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.140504 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.140392 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-secret-alertmanager-kube-rbac-proxy-web\") pod \"0cedefa7-efaf-490b-b9b8-e3fb34570367\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " Jul 27 20:37:08.140504 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.140439 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0cedefa7-efaf-490b-b9b8-e3fb34570367-alertmanager-main-db\") pod \"0cedefa7-efaf-490b-b9b8-e3fb34570367\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " Jul 27 20:37:08.140504 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.140457 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-secret-alertmanager-main-tls\") pod \"0cedefa7-efaf-490b-b9b8-e3fb34570367\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " Jul 27 20:37:08.140504 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.140496 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0cedefa7-efaf-490b-b9b8-e3fb34570367-metrics-client-ca\") pod \"0cedefa7-efaf-490b-b9b8-e3fb34570367\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " Jul 27 20:37:08.140850 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.140531 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0cedefa7-efaf-490b-b9b8-e3fb34570367-config-out\") pod \"0cedefa7-efaf-490b-b9b8-e3fb34570367\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " Jul 27 20:37:08.140850 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.140562 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-secret-alertmanager-kube-rbac-proxy-metric\") pod \"0cedefa7-efaf-490b-b9b8-e3fb34570367\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " Jul 27 20:37:08.140850 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.140580 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-config-volume\") pod \"0cedefa7-efaf-490b-b9b8-e3fb34570367\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " Jul 27 20:37:08.140850 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.140644 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-web-config\") pod \"0cedefa7-efaf-490b-b9b8-e3fb34570367\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " Jul 27 20:37:08.141050 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.141004 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fd7n\" (UniqueName: \"kubernetes.io/projected/0cedefa7-efaf-490b-b9b8-e3fb34570367-kube-api-access-9fd7n\") pod \"0cedefa7-efaf-490b-b9b8-e3fb34570367\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " Jul 27 20:37:08.141094 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.141063 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-secret-alertmanager-kube-rbac-proxy\") pod \"0cedefa7-efaf-490b-b9b8-e3fb34570367\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " Jul 27 20:37:08.141144 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.141123 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cedefa7-efaf-490b-b9b8-e3fb34570367-alertmanager-trusted-ca-bundle\") pod \"0cedefa7-efaf-490b-b9b8-e3fb34570367\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " Jul 27 20:37:08.141187 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.141165 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-cluster-tls-config\") pod \"0cedefa7-efaf-490b-b9b8-e3fb34570367\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " Jul 27 20:37:08.141233 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.141209 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0cedefa7-efaf-490b-b9b8-e3fb34570367-tls-assets\") pod \"0cedefa7-efaf-490b-b9b8-e3fb34570367\" (UID: \"0cedefa7-efaf-490b-b9b8-e3fb34570367\") " Jul 27 20:37:08.141289 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.141226 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cedefa7-efaf-490b-b9b8-e3fb34570367-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "0cedefa7-efaf-490b-b9b8-e3fb34570367" (UID: "0cedefa7-efaf-490b-b9b8-e3fb34570367"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:37:08.141346 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.141285 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cedefa7-efaf-490b-b9b8-e3fb34570367-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "0cedefa7-efaf-490b-b9b8-e3fb34570367" (UID: "0cedefa7-efaf-490b-b9b8-e3fb34570367"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 20:37:08.142522 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.142080 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cedefa7-efaf-490b-b9b8-e3fb34570367-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "0cedefa7-efaf-490b-b9b8-e3fb34570367" (UID: "0cedefa7-efaf-490b-b9b8-e3fb34570367"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:37:08.143242 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.143175 2570 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0cedefa7-efaf-490b-b9b8-e3fb34570367-alertmanager-main-db\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:37:08.143242 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.143206 2570 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0cedefa7-efaf-490b-b9b8-e3fb34570367-metrics-client-ca\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:37:08.149566 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.149482 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "0cedefa7-efaf-490b-b9b8-e3fb34570367" (UID: "0cedefa7-efaf-490b-b9b8-e3fb34570367"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 20:37:08.150455 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.150334 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cedefa7-efaf-490b-b9b8-e3fb34570367-config-out" (OuterVolumeSpecName: "config-out") pod "0cedefa7-efaf-490b-b9b8-e3fb34570367" (UID: "0cedefa7-efaf-490b-b9b8-e3fb34570367"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 20:37:08.150455 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.150345 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cedefa7-efaf-490b-b9b8-e3fb34570367-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "0cedefa7-efaf-490b-b9b8-e3fb34570367" (UID: "0cedefa7-efaf-490b-b9b8-e3fb34570367"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:37:08.150825 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.150799 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "0cedefa7-efaf-490b-b9b8-e3fb34570367" (UID: "0cedefa7-efaf-490b-b9b8-e3fb34570367"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 20:37:08.150966 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.150942 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "0cedefa7-efaf-490b-b9b8-e3fb34570367" (UID: "0cedefa7-efaf-490b-b9b8-e3fb34570367"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 20:37:08.151438 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.151411 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "0cedefa7-efaf-490b-b9b8-e3fb34570367" (UID: "0cedefa7-efaf-490b-b9b8-e3fb34570367"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 20:37:08.151591 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.151509 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cedefa7-efaf-490b-b9b8-e3fb34570367-kube-api-access-9fd7n" (OuterVolumeSpecName: "kube-api-access-9fd7n") pod "0cedefa7-efaf-490b-b9b8-e3fb34570367" (UID: "0cedefa7-efaf-490b-b9b8-e3fb34570367"). InnerVolumeSpecName "kube-api-access-9fd7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:37:08.152579 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.152554 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-config-volume" (OuterVolumeSpecName: "config-volume") pod "0cedefa7-efaf-490b-b9b8-e3fb34570367" (UID: "0cedefa7-efaf-490b-b9b8-e3fb34570367"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 20:37:08.154579 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.154559 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "0cedefa7-efaf-490b-b9b8-e3fb34570367" (UID: "0cedefa7-efaf-490b-b9b8-e3fb34570367"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 20:37:08.160750 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.160726 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-web-config" (OuterVolumeSpecName: "web-config") pod "0cedefa7-efaf-490b-b9b8-e3fb34570367" (UID: "0cedefa7-efaf-490b-b9b8-e3fb34570367"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 20:37:08.243614 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.243573 2570 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cedefa7-efaf-490b-b9b8-e3fb34570367-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:37:08.243793 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.243666 2570 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-cluster-tls-config\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:37:08.243793 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.243681 2570 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0cedefa7-efaf-490b-b9b8-e3fb34570367-tls-assets\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:37:08.243793 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.243694 2570 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:37:08.243793 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.243706 2570 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-secret-alertmanager-main-tls\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:37:08.243793 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.243717 2570 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0cedefa7-efaf-490b-b9b8-e3fb34570367-config-out\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:37:08.243793 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.243728 2570 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:37:08.243793 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.243742 2570 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-config-volume\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:37:08.243793 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.243754 2570 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-web-config\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:37:08.243793 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.243765 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fd7n\" (UniqueName: \"kubernetes.io/projected/0cedefa7-efaf-490b-b9b8-e3fb34570367-kube-api-access-9fd7n\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:37:08.243793 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.243777 2570 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0cedefa7-efaf-490b-b9b8-e3fb34570367-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:37:08.594951 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.594912 2570 generic.go:356] "Generic (PLEG): container finished" podID="0cedefa7-efaf-490b-b9b8-e3fb34570367" containerID="2065d07b04d958742261ba63f2a6564b91443f9a18330fdd59b7eea6ca0ad5f2" exitCode=0 Jul 27 20:37:08.595149 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.594981 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0cedefa7-efaf-490b-b9b8-e3fb34570367","Type":"ContainerDied","Data":"2065d07b04d958742261ba63f2a6564b91443f9a18330fdd59b7eea6ca0ad5f2"} Jul 27 20:37:08.595149 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.595057 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.595149 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.595096 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0cedefa7-efaf-490b-b9b8-e3fb34570367","Type":"ContainerDied","Data":"17d7e2a30ec33067cdbf89ab2c2e0ebb3c000edcbfb2fdd0a700c39082345dc5"} Jul 27 20:37:08.595149 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.595145 2570 scope.go:117] "RemoveContainer" containerID="80b02820c600b83907d7537b98b951b6e1b2f550e823b8d4298c4af37e0ecaea" Jul 27 20:37:08.603116 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.603094 2570 scope.go:117] "RemoveContainer" containerID="256392deb3ad2d9faeb83790f4f41b68f40e834bd841c6458695295b029d430b" Jul 27 20:37:08.610648 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.610628 2570 scope.go:117] "RemoveContainer" containerID="46c79ce1d1071a51103771c8b5c6ba15fdf5603efa41f243442ed930bb83069c" Jul 27 20:37:08.615912 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.615888 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 20:37:08.618768 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.618653 2570 scope.go:117] "RemoveContainer" containerID="2065d07b04d958742261ba63f2a6564b91443f9a18330fdd59b7eea6ca0ad5f2" Jul 27 20:37:08.620791 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.620769 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 20:37:08.626153 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.626134 2570 scope.go:117] "RemoveContainer" containerID="8358940e2b1c377288038a3be9f75f0b3d9d9693b5835afd66ba90a8d456da97" Jul 27 20:37:08.633324 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.633303 2570 scope.go:117] "RemoveContainer" containerID="2e08a7fe270f49c4181868ab8af97cfe14235bcc7298ee83c443845ec7986e86" Jul 27 20:37:08.640771 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.640749 2570 scope.go:117] "RemoveContainer" containerID="7a5920ea941e129697a6a269970ed9ab85bebf79f858f958376bae983c668e27" Jul 27 20:37:08.642393 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.642375 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 20:37:08.642692 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.642678 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cedefa7-efaf-490b-b9b8-e3fb34570367" containerName="kube-rbac-proxy-metric" Jul 27 20:37:08.642742 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.642693 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cedefa7-efaf-490b-b9b8-e3fb34570367" containerName="kube-rbac-proxy-metric" Jul 27 20:37:08.642742 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.642701 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cedefa7-efaf-490b-b9b8-e3fb34570367" containerName="kube-rbac-proxy" Jul 27 20:37:08.642742 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.642707 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cedefa7-efaf-490b-b9b8-e3fb34570367" containerName="kube-rbac-proxy" Jul 27 20:37:08.642742 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.642712 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cedefa7-efaf-490b-b9b8-e3fb34570367" containerName="prom-label-proxy" Jul 27 20:37:08.642742 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.642718 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cedefa7-efaf-490b-b9b8-e3fb34570367" containerName="prom-label-proxy" Jul 27 20:37:08.642742 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.642725 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cedefa7-efaf-490b-b9b8-e3fb34570367" containerName="config-reloader" Jul 27 20:37:08.642742 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.642730 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cedefa7-efaf-490b-b9b8-e3fb34570367" containerName="config-reloader" Jul 27 20:37:08.642742 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.642737 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cedefa7-efaf-490b-b9b8-e3fb34570367" containerName="alertmanager" Jul 27 20:37:08.642742 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.642742 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cedefa7-efaf-490b-b9b8-e3fb34570367" containerName="alertmanager" Jul 27 20:37:08.643061 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.642748 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cedefa7-efaf-490b-b9b8-e3fb34570367" containerName="kube-rbac-proxy-web" Jul 27 20:37:08.643061 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.642753 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cedefa7-efaf-490b-b9b8-e3fb34570367" containerName="kube-rbac-proxy-web" Jul 27 20:37:08.643061 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.642762 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d90ced29-8d27-4cd9-9ec1-a6e7132a1365" containerName="console" Jul 27 20:37:08.643061 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.642767 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d90ced29-8d27-4cd9-9ec1-a6e7132a1365" containerName="console" Jul 27 20:37:08.643061 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.642775 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cedefa7-efaf-490b-b9b8-e3fb34570367" containerName="init-config-reloader" Jul 27 20:37:08.643061 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.642780 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cedefa7-efaf-490b-b9b8-e3fb34570367" containerName="init-config-reloader" Jul 27 20:37:08.643061 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.642821 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d90ced29-8d27-4cd9-9ec1-a6e7132a1365" containerName="console" Jul 27 20:37:08.643061 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.642828 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cedefa7-efaf-490b-b9b8-e3fb34570367" containerName="config-reloader" Jul 27 20:37:08.643061 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.642834 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cedefa7-efaf-490b-b9b8-e3fb34570367" containerName="alertmanager" Jul 27 20:37:08.643061 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.642841 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cedefa7-efaf-490b-b9b8-e3fb34570367" containerName="kube-rbac-proxy-metric" Jul 27 20:37:08.643061 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.642846 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cedefa7-efaf-490b-b9b8-e3fb34570367" containerName="prom-label-proxy" Jul 27 20:37:08.643061 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.642851 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cedefa7-efaf-490b-b9b8-e3fb34570367" containerName="kube-rbac-proxy-web" Jul 27 20:37:08.643061 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.642856 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cedefa7-efaf-490b-b9b8-e3fb34570367" containerName="kube-rbac-proxy" Jul 27 20:37:08.648420 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.647997 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.650043 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.650023 2570 scope.go:117] "RemoveContainer" containerID="80b02820c600b83907d7537b98b951b6e1b2f550e823b8d4298c4af37e0ecaea" Jul 27 20:37:08.650346 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:37:08.650329 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80b02820c600b83907d7537b98b951b6e1b2f550e823b8d4298c4af37e0ecaea\": container with ID starting with 80b02820c600b83907d7537b98b951b6e1b2f550e823b8d4298c4af37e0ecaea not found: ID does not exist" containerID="80b02820c600b83907d7537b98b951b6e1b2f550e823b8d4298c4af37e0ecaea" Jul 27 20:37:08.650401 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.650357 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80b02820c600b83907d7537b98b951b6e1b2f550e823b8d4298c4af37e0ecaea"} err="failed to get container status \"80b02820c600b83907d7537b98b951b6e1b2f550e823b8d4298c4af37e0ecaea\": rpc error: code = NotFound desc = could not find container \"80b02820c600b83907d7537b98b951b6e1b2f550e823b8d4298c4af37e0ecaea\": container with ID starting with 80b02820c600b83907d7537b98b951b6e1b2f550e823b8d4298c4af37e0ecaea not found: ID does not exist" Jul 27 20:37:08.650401 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.650377 2570 scope.go:117] "RemoveContainer" containerID="256392deb3ad2d9faeb83790f4f41b68f40e834bd841c6458695295b029d430b" Jul 27 20:37:08.650654 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:37:08.650632 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"256392deb3ad2d9faeb83790f4f41b68f40e834bd841c6458695295b029d430b\": container with ID starting with 256392deb3ad2d9faeb83790f4f41b68f40e834bd841c6458695295b029d430b not found: ID does not exist" containerID="256392deb3ad2d9faeb83790f4f41b68f40e834bd841c6458695295b029d430b" Jul 27 20:37:08.650715 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.650657 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"256392deb3ad2d9faeb83790f4f41b68f40e834bd841c6458695295b029d430b"} err="failed to get container status \"256392deb3ad2d9faeb83790f4f41b68f40e834bd841c6458695295b029d430b\": rpc error: code = NotFound desc = could not find container \"256392deb3ad2d9faeb83790f4f41b68f40e834bd841c6458695295b029d430b\": container with ID starting with 256392deb3ad2d9faeb83790f4f41b68f40e834bd841c6458695295b029d430b not found: ID does not exist" Jul 27 20:37:08.650715 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.650670 2570 scope.go:117] "RemoveContainer" containerID="46c79ce1d1071a51103771c8b5c6ba15fdf5603efa41f243442ed930bb83069c" Jul 27 20:37:08.650839 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.650817 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 27 20:37:08.650839 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.650835 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 27 20:37:08.650987 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.650925 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 27 20:37:08.650987 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:37:08.650935 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46c79ce1d1071a51103771c8b5c6ba15fdf5603efa41f243442ed930bb83069c\": container with ID starting with 46c79ce1d1071a51103771c8b5c6ba15fdf5603efa41f243442ed930bb83069c not found: ID does not exist" containerID="46c79ce1d1071a51103771c8b5c6ba15fdf5603efa41f243442ed930bb83069c" Jul 27 20:37:08.650987 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.650961 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46c79ce1d1071a51103771c8b5c6ba15fdf5603efa41f243442ed930bb83069c"} err="failed to get container status \"46c79ce1d1071a51103771c8b5c6ba15fdf5603efa41f243442ed930bb83069c\": rpc error: code = NotFound desc = could not find container \"46c79ce1d1071a51103771c8b5c6ba15fdf5603efa41f243442ed930bb83069c\": container with ID starting with 46c79ce1d1071a51103771c8b5c6ba15fdf5603efa41f243442ed930bb83069c not found: ID does not exist" Jul 27 20:37:08.650987 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.650982 2570 scope.go:117] "RemoveContainer" containerID="2065d07b04d958742261ba63f2a6564b91443f9a18330fdd59b7eea6ca0ad5f2" Jul 27 20:37:08.651192 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.651005 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 27 20:37:08.651294 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.651271 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 27 20:37:08.651390 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.651295 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 27 20:37:08.651390 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.651338 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 27 20:37:08.651390 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.651369 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-dc8tf\"" Jul 27 20:37:08.651390 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:37:08.651335 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2065d07b04d958742261ba63f2a6564b91443f9a18330fdd59b7eea6ca0ad5f2\": container with ID starting with 2065d07b04d958742261ba63f2a6564b91443f9a18330fdd59b7eea6ca0ad5f2 not found: ID does not exist" containerID="2065d07b04d958742261ba63f2a6564b91443f9a18330fdd59b7eea6ca0ad5f2" Jul 27 20:37:08.651592 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.651401 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2065d07b04d958742261ba63f2a6564b91443f9a18330fdd59b7eea6ca0ad5f2"} err="failed to get container status \"2065d07b04d958742261ba63f2a6564b91443f9a18330fdd59b7eea6ca0ad5f2\": rpc error: code = NotFound desc = could not find container \"2065d07b04d958742261ba63f2a6564b91443f9a18330fdd59b7eea6ca0ad5f2\": container with ID starting with 2065d07b04d958742261ba63f2a6564b91443f9a18330fdd59b7eea6ca0ad5f2 not found: ID does not exist" Jul 27 20:37:08.651592 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.651421 2570 scope.go:117] "RemoveContainer" containerID="8358940e2b1c377288038a3be9f75f0b3d9d9693b5835afd66ba90a8d456da97" Jul 27 20:37:08.651592 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.651378 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 27 20:37:08.651809 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:37:08.651730 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8358940e2b1c377288038a3be9f75f0b3d9d9693b5835afd66ba90a8d456da97\": container with ID starting with 8358940e2b1c377288038a3be9f75f0b3d9d9693b5835afd66ba90a8d456da97 not found: ID does not exist" containerID="8358940e2b1c377288038a3be9f75f0b3d9d9693b5835afd66ba90a8d456da97" Jul 27 20:37:08.651809 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.651753 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8358940e2b1c377288038a3be9f75f0b3d9d9693b5835afd66ba90a8d456da97"} err="failed to get container status \"8358940e2b1c377288038a3be9f75f0b3d9d9693b5835afd66ba90a8d456da97\": rpc error: code = NotFound desc = could not find container \"8358940e2b1c377288038a3be9f75f0b3d9d9693b5835afd66ba90a8d456da97\": container with ID starting with 8358940e2b1c377288038a3be9f75f0b3d9d9693b5835afd66ba90a8d456da97 not found: ID does not exist" Jul 27 20:37:08.651809 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.651773 2570 scope.go:117] "RemoveContainer" containerID="2e08a7fe270f49c4181868ab8af97cfe14235bcc7298ee83c443845ec7986e86" Jul 27 20:37:08.652181 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:37:08.651985 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e08a7fe270f49c4181868ab8af97cfe14235bcc7298ee83c443845ec7986e86\": container with ID starting with 2e08a7fe270f49c4181868ab8af97cfe14235bcc7298ee83c443845ec7986e86 not found: ID does not exist" containerID="2e08a7fe270f49c4181868ab8af97cfe14235bcc7298ee83c443845ec7986e86" Jul 27 20:37:08.652181 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.652007 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e08a7fe270f49c4181868ab8af97cfe14235bcc7298ee83c443845ec7986e86"} err="failed to get container status \"2e08a7fe270f49c4181868ab8af97cfe14235bcc7298ee83c443845ec7986e86\": rpc error: code = NotFound desc = could not find container \"2e08a7fe270f49c4181868ab8af97cfe14235bcc7298ee83c443845ec7986e86\": container with ID starting with 2e08a7fe270f49c4181868ab8af97cfe14235bcc7298ee83c443845ec7986e86 not found: ID does not exist" Jul 27 20:37:08.652181 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.652029 2570 scope.go:117] "RemoveContainer" containerID="7a5920ea941e129697a6a269970ed9ab85bebf79f858f958376bae983c668e27" Jul 27 20:37:08.653069 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:37:08.653050 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a5920ea941e129697a6a269970ed9ab85bebf79f858f958376bae983c668e27\": container with ID starting with 7a5920ea941e129697a6a269970ed9ab85bebf79f858f958376bae983c668e27 not found: ID does not exist" containerID="7a5920ea941e129697a6a269970ed9ab85bebf79f858f958376bae983c668e27" Jul 27 20:37:08.653169 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.653072 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a5920ea941e129697a6a269970ed9ab85bebf79f858f958376bae983c668e27"} err="failed to get container status \"7a5920ea941e129697a6a269970ed9ab85bebf79f858f958376bae983c668e27\": rpc error: code = NotFound desc = could not find container \"7a5920ea941e129697a6a269970ed9ab85bebf79f858f958376bae983c668e27\": container with ID starting with 7a5920ea941e129697a6a269970ed9ab85bebf79f858f958376bae983c668e27 not found: ID does not exist" Jul 27 20:37:08.657224 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.657207 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 27 20:37:08.665348 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.665322 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 20:37:08.747248 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.747208 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-web-config\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.747248 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.747249 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krbf2\" (UniqueName: \"kubernetes.io/projected/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-kube-api-access-krbf2\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.747455 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.747272 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.747455 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.747299 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.747455 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.747319 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-config-volume\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.747455 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.747332 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.747455 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.747353 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.747455 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.747395 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.747455 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.747418 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.747455 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.747438 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.747455 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.747455 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.747775 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.747473 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.747775 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.747491 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-config-out\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.847921 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.847825 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.847921 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.847876 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.847921 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.847904 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.848191 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.847937 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-config-out\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.848191 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.847967 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-web-config\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.848191 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.848147 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krbf2\" (UniqueName: \"kubernetes.io/projected/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-kube-api-access-krbf2\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.848191 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.848188 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.848386 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.848222 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.848386 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.848253 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-config-volume\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.848386 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.848274 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.848386 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.848309 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.848386 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.848343 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.848828 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.848390 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.849456 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.849429 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.852231 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.851257 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-config-out\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.852231 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.851301 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.852231 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.851312 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.852231 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.851416 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.852231 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.851775 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-config-volume\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.852231 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.852182 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.852231 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.852231 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-web-config\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.852692 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.852258 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.852692 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.852496 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.853659 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.853584 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.853945 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.853929 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.856135 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.856111 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cedefa7-efaf-490b-b9b8-e3fb34570367" path="/var/lib/kubelet/pods/0cedefa7-efaf-490b-b9b8-e3fb34570367/volumes" Jul 27 20:37:08.859875 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.859852 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krbf2\" (UniqueName: \"kubernetes.io/projected/5f3715d7-d32e-4c67-b57a-a91e4df5fa22-kube-api-access-krbf2\") pod \"alertmanager-main-0\" (UID: \"5f3715d7-d32e-4c67-b57a-a91e4df5fa22\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:08.959029 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:08.958992 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 27 20:37:09.098126 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:09.098077 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 20:37:09.100091 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:37:09.100062 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f3715d7_d32e_4c67_b57a_a91e4df5fa22.slice/crio-cee7ddcf3648fafd80c97a1219c90a35566248bdf990998e1877d20de398c315 WatchSource:0}: Error finding container cee7ddcf3648fafd80c97a1219c90a35566248bdf990998e1877d20de398c315: Status 404 returned error can't find the container with id cee7ddcf3648fafd80c97a1219c90a35566248bdf990998e1877d20de398c315 Jul 27 20:37:09.601070 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:09.601032 2570 generic.go:356] "Generic (PLEG): container finished" podID="5f3715d7-d32e-4c67-b57a-a91e4df5fa22" containerID="f4eb964bf299b83179aeb93d717e518683d70bae769a283469415ed6bcc399e0" exitCode=0 Jul 27 20:37:09.601234 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:09.601094 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f3715d7-d32e-4c67-b57a-a91e4df5fa22","Type":"ContainerDied","Data":"f4eb964bf299b83179aeb93d717e518683d70bae769a283469415ed6bcc399e0"} Jul 27 20:37:09.601234 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:09.601164 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f3715d7-d32e-4c67-b57a-a91e4df5fa22","Type":"ContainerStarted","Data":"cee7ddcf3648fafd80c97a1219c90a35566248bdf990998e1877d20de398c315"} Jul 27 20:37:10.607935 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.607895 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f3715d7-d32e-4c67-b57a-a91e4df5fa22","Type":"ContainerStarted","Data":"42b169f28eab6e8ca269937e51960b6f05bdd26a2d6f38ea82cac062d9a0d811"} Jul 27 20:37:10.608362 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.607997 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f3715d7-d32e-4c67-b57a-a91e4df5fa22","Type":"ContainerStarted","Data":"a46e0ebfb45d1c54befb9288c92fac683d54df3916ee8149bd3213f9880a3f9c"} Jul 27 20:37:10.608362 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.608028 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f3715d7-d32e-4c67-b57a-a91e4df5fa22","Type":"ContainerStarted","Data":"eab16c329e9a93a017d5f810c23ac4fb302621f13b48392968a4361fcf1b1234"} Jul 27 20:37:10.608362 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.608056 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f3715d7-d32e-4c67-b57a-a91e4df5fa22","Type":"ContainerStarted","Data":"fd1f57a549da4a840c74c511ebb6c38da4fa674ffdd29e910de348efb856c24f"} Jul 27 20:37:10.608362 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.608088 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f3715d7-d32e-4c67-b57a-a91e4df5fa22","Type":"ContainerStarted","Data":"16487669441d3be6dcae05f3120416baecdef87f5568c12049e968cc33e6cddc"} Jul 27 20:37:10.608362 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.608112 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f3715d7-d32e-4c67-b57a-a91e4df5fa22","Type":"ContainerStarted","Data":"fed7d67c887d3e1ea71629d55d51b81c7a7a203633aba1d2f8a54fd801bbdf43"} Jul 27 20:37:10.633922 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.633860 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.618103583 podStartE2EDuration="2.633842974s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.739391ms" isStatefulPod=false podCreationTimestamp="2026-07-27 20:37:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:37:10.632295518 +0000 UTC m=+160.368275805" watchObservedRunningTime="2026-07-27 20:37:10.633842974 +0000 UTC m=+160.369823271" Jul 27 20:37:10.741854 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.741816 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-cb749fc9-pgnpl"] Jul 27 20:37:10.745022 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.744998 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-cb749fc9-pgnpl" Jul 27 20:37:10.748375 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.747757 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 27 20:37:10.748375 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.747867 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 27 20:37:10.748375 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.747980 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 27 20:37:10.748375 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.748262 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 27 20:37:10.748375 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.748294 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-sx688\"" Jul 27 20:37:10.748728 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.748475 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 27 20:37:10.753933 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.753905 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 27 20:37:10.757426 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.757397 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-cb749fc9-pgnpl"] Jul 27 20:37:10.764373 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.763881 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5374772e-877c-4718-8ae1-7e397424d3bc-telemeter-client-tls\") pod \"telemeter-client-cb749fc9-pgnpl\" (UID: \"5374772e-877c-4718-8ae1-7e397424d3bc\") " pod="openshift-monitoring/telemeter-client-cb749fc9-pgnpl" Jul 27 20:37:10.764373 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.763940 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5374772e-877c-4718-8ae1-7e397424d3bc-telemeter-trusted-ca-bundle\") pod \"telemeter-client-cb749fc9-pgnpl\" (UID: \"5374772e-877c-4718-8ae1-7e397424d3bc\") " pod="openshift-monitoring/telemeter-client-cb749fc9-pgnpl" Jul 27 20:37:10.764373 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.764002 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5374772e-877c-4718-8ae1-7e397424d3bc-metrics-client-ca\") pod \"telemeter-client-cb749fc9-pgnpl\" (UID: \"5374772e-877c-4718-8ae1-7e397424d3bc\") " pod="openshift-monitoring/telemeter-client-cb749fc9-pgnpl" Jul 27 20:37:10.764373 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.764047 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5374772e-877c-4718-8ae1-7e397424d3bc-secret-telemeter-client\") pod \"telemeter-client-cb749fc9-pgnpl\" (UID: \"5374772e-877c-4718-8ae1-7e397424d3bc\") " pod="openshift-monitoring/telemeter-client-cb749fc9-pgnpl" Jul 27 20:37:10.764373 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.764075 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5374772e-877c-4718-8ae1-7e397424d3bc-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-cb749fc9-pgnpl\" (UID: \"5374772e-877c-4718-8ae1-7e397424d3bc\") " pod="openshift-monitoring/telemeter-client-cb749fc9-pgnpl" Jul 27 20:37:10.764373 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.764119 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5374772e-877c-4718-8ae1-7e397424d3bc-federate-client-tls\") pod \"telemeter-client-cb749fc9-pgnpl\" (UID: \"5374772e-877c-4718-8ae1-7e397424d3bc\") " pod="openshift-monitoring/telemeter-client-cb749fc9-pgnpl" Jul 27 20:37:10.764373 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.764148 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5374772e-877c-4718-8ae1-7e397424d3bc-serving-certs-ca-bundle\") pod \"telemeter-client-cb749fc9-pgnpl\" (UID: \"5374772e-877c-4718-8ae1-7e397424d3bc\") " pod="openshift-monitoring/telemeter-client-cb749fc9-pgnpl" Jul 27 20:37:10.764373 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.764179 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6n9r\" (UniqueName: \"kubernetes.io/projected/5374772e-877c-4718-8ae1-7e397424d3bc-kube-api-access-z6n9r\") pod \"telemeter-client-cb749fc9-pgnpl\" (UID: \"5374772e-877c-4718-8ae1-7e397424d3bc\") " pod="openshift-monitoring/telemeter-client-cb749fc9-pgnpl" Jul 27 20:37:10.864661 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.864535 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5374772e-877c-4718-8ae1-7e397424d3bc-serving-certs-ca-bundle\") pod \"telemeter-client-cb749fc9-pgnpl\" (UID: \"5374772e-877c-4718-8ae1-7e397424d3bc\") " pod="openshift-monitoring/telemeter-client-cb749fc9-pgnpl" Jul 27 20:37:10.864661 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.864582 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6n9r\" (UniqueName: \"kubernetes.io/projected/5374772e-877c-4718-8ae1-7e397424d3bc-kube-api-access-z6n9r\") pod \"telemeter-client-cb749fc9-pgnpl\" (UID: \"5374772e-877c-4718-8ae1-7e397424d3bc\") " pod="openshift-monitoring/telemeter-client-cb749fc9-pgnpl" Jul 27 20:37:10.864661 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.864636 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5374772e-877c-4718-8ae1-7e397424d3bc-telemeter-client-tls\") pod \"telemeter-client-cb749fc9-pgnpl\" (UID: \"5374772e-877c-4718-8ae1-7e397424d3bc\") " pod="openshift-monitoring/telemeter-client-cb749fc9-pgnpl" Jul 27 20:37:10.864931 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.864678 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5374772e-877c-4718-8ae1-7e397424d3bc-telemeter-trusted-ca-bundle\") pod \"telemeter-client-cb749fc9-pgnpl\" (UID: \"5374772e-877c-4718-8ae1-7e397424d3bc\") " pod="openshift-monitoring/telemeter-client-cb749fc9-pgnpl" Jul 27 20:37:10.864931 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.864713 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5374772e-877c-4718-8ae1-7e397424d3bc-metrics-client-ca\") pod \"telemeter-client-cb749fc9-pgnpl\" (UID: \"5374772e-877c-4718-8ae1-7e397424d3bc\") " pod="openshift-monitoring/telemeter-client-cb749fc9-pgnpl" Jul 27 20:37:10.864931 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.864753 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5374772e-877c-4718-8ae1-7e397424d3bc-secret-telemeter-client\") pod \"telemeter-client-cb749fc9-pgnpl\" (UID: \"5374772e-877c-4718-8ae1-7e397424d3bc\") " pod="openshift-monitoring/telemeter-client-cb749fc9-pgnpl" Jul 27 20:37:10.864931 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.864780 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5374772e-877c-4718-8ae1-7e397424d3bc-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-cb749fc9-pgnpl\" (UID: \"5374772e-877c-4718-8ae1-7e397424d3bc\") " pod="openshift-monitoring/telemeter-client-cb749fc9-pgnpl" Jul 27 20:37:10.865096 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.865075 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5374772e-877c-4718-8ae1-7e397424d3bc-federate-client-tls\") pod \"telemeter-client-cb749fc9-pgnpl\" (UID: \"5374772e-877c-4718-8ae1-7e397424d3bc\") " pod="openshift-monitoring/telemeter-client-cb749fc9-pgnpl" Jul 27 20:37:10.865542 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.865490 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5374772e-877c-4718-8ae1-7e397424d3bc-serving-certs-ca-bundle\") pod \"telemeter-client-cb749fc9-pgnpl\" (UID: \"5374772e-877c-4718-8ae1-7e397424d3bc\") " pod="openshift-monitoring/telemeter-client-cb749fc9-pgnpl" Jul 27 20:37:10.865676 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.865634 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5374772e-877c-4718-8ae1-7e397424d3bc-metrics-client-ca\") pod \"telemeter-client-cb749fc9-pgnpl\" (UID: \"5374772e-877c-4718-8ae1-7e397424d3bc\") " pod="openshift-monitoring/telemeter-client-cb749fc9-pgnpl" Jul 27 20:37:10.865899 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.865678 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5374772e-877c-4718-8ae1-7e397424d3bc-telemeter-trusted-ca-bundle\") pod \"telemeter-client-cb749fc9-pgnpl\" (UID: \"5374772e-877c-4718-8ae1-7e397424d3bc\") " pod="openshift-monitoring/telemeter-client-cb749fc9-pgnpl" Jul 27 20:37:10.867653 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.867633 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5374772e-877c-4718-8ae1-7e397424d3bc-federate-client-tls\") pod \"telemeter-client-cb749fc9-pgnpl\" (UID: \"5374772e-877c-4718-8ae1-7e397424d3bc\") " pod="openshift-monitoring/telemeter-client-cb749fc9-pgnpl" Jul 27 20:37:10.867983 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.867958 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5374772e-877c-4718-8ae1-7e397424d3bc-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-cb749fc9-pgnpl\" (UID: \"5374772e-877c-4718-8ae1-7e397424d3bc\") " pod="openshift-monitoring/telemeter-client-cb749fc9-pgnpl" Jul 27 20:37:10.868058 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.867964 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5374772e-877c-4718-8ae1-7e397424d3bc-secret-telemeter-client\") pod \"telemeter-client-cb749fc9-pgnpl\" (UID: \"5374772e-877c-4718-8ae1-7e397424d3bc\") " pod="openshift-monitoring/telemeter-client-cb749fc9-pgnpl" Jul 27 20:37:10.868058 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.867969 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5374772e-877c-4718-8ae1-7e397424d3bc-telemeter-client-tls\") pod \"telemeter-client-cb749fc9-pgnpl\" (UID: \"5374772e-877c-4718-8ae1-7e397424d3bc\") " pod="openshift-monitoring/telemeter-client-cb749fc9-pgnpl" Jul 27 20:37:10.873995 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:10.873962 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6n9r\" (UniqueName: \"kubernetes.io/projected/5374772e-877c-4718-8ae1-7e397424d3bc-kube-api-access-z6n9r\") pod \"telemeter-client-cb749fc9-pgnpl\" (UID: \"5374772e-877c-4718-8ae1-7e397424d3bc\") " pod="openshift-monitoring/telemeter-client-cb749fc9-pgnpl" Jul 27 20:37:11.058416 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:11.058372 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-cb749fc9-pgnpl" Jul 27 20:37:11.224383 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:11.224358 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-cb749fc9-pgnpl"] Jul 27 20:37:11.226841 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:37:11.226814 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5374772e_877c_4718_8ae1_7e397424d3bc.slice/crio-2db8780d12a4d8d6c1d803a21135706b4dd1a3639c9b7b2c25bc896e770d1fd7 WatchSource:0}: Error finding container 2db8780d12a4d8d6c1d803a21135706b4dd1a3639c9b7b2c25bc896e770d1fd7: Status 404 returned error can't find the container with id 2db8780d12a4d8d6c1d803a21135706b4dd1a3639c9b7b2c25bc896e770d1fd7 Jul 27 20:37:11.612699 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:11.612667 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-cb749fc9-pgnpl" event={"ID":"5374772e-877c-4718-8ae1-7e397424d3bc","Type":"ContainerStarted","Data":"2db8780d12a4d8d6c1d803a21135706b4dd1a3639c9b7b2c25bc896e770d1fd7"} Jul 27 20:37:13.623673 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:13.623567 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-cb749fc9-pgnpl" event={"ID":"5374772e-877c-4718-8ae1-7e397424d3bc","Type":"ContainerStarted","Data":"d9a7ad437bb8fe499ff173b2a12b544cb5dfb24d1a26785e0c1edae23cb1b582"} Jul 27 20:37:13.623673 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:13.623656 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-cb749fc9-pgnpl" event={"ID":"5374772e-877c-4718-8ae1-7e397424d3bc","Type":"ContainerStarted","Data":"577fef0a32fc55b04e977666650b2f811d1154f58a6dbf704155157285fd119a"} Jul 27 20:37:13.624222 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:13.623687 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-cb749fc9-pgnpl" event={"ID":"5374772e-877c-4718-8ae1-7e397424d3bc","Type":"ContainerStarted","Data":"5c2cd3d49b29333c8f605baaf95397289d7c3bce5d831d66fca6ad694d02bd13"} Jul 27 20:37:13.649884 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:13.649830 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-cb749fc9-pgnpl" podStartSLOduration=1.582864547 podStartE2EDuration="3.649809795s" totalImagesPullingTime="2.066945248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:37:13.649324196 +0000 UTC m=+163.385304486" watchObservedRunningTime="2026-07-27 20:37:13.649809795 +0000 UTC m=+163.385790084" Jul 27 20:37:14.549414 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:14.549378 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-dc87ddb86-5gm4v"] Jul 27 20:37:14.551888 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:14.551858 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dc87ddb86-5gm4v" Jul 27 20:37:14.554720 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:14.554697 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 27 20:37:14.554838 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:14.554701 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 27 20:37:14.554973 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:14.554956 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 27 20:37:14.555063 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:14.555001 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 27 20:37:14.555063 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:14.555056 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 27 20:37:14.555936 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:14.555918 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 27 20:37:14.556050 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:14.555982 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 27 20:37:14.556107 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:14.556078 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rmhbg\"" Jul 27 20:37:14.560026 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:14.559999 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 27 20:37:14.563118 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:14.563094 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dc87ddb86-5gm4v"] Jul 27 20:37:14.598588 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:14.598555 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f993f43-1925-403a-9fd5-2d7e9e1159f9-trusted-ca-bundle\") pod \"console-dc87ddb86-5gm4v\" (UID: \"9f993f43-1925-403a-9fd5-2d7e9e1159f9\") " pod="openshift-console/console-dc87ddb86-5gm4v" Jul 27 20:37:14.598588 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:14.598594 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f993f43-1925-403a-9fd5-2d7e9e1159f9-console-serving-cert\") pod \"console-dc87ddb86-5gm4v\" (UID: \"9f993f43-1925-403a-9fd5-2d7e9e1159f9\") " pod="openshift-console/console-dc87ddb86-5gm4v" Jul 27 20:37:14.598806 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:14.598629 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f993f43-1925-403a-9fd5-2d7e9e1159f9-oauth-serving-cert\") pod \"console-dc87ddb86-5gm4v\" (UID: \"9f993f43-1925-403a-9fd5-2d7e9e1159f9\") " pod="openshift-console/console-dc87ddb86-5gm4v" Jul 27 20:37:14.598806 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:14.598709 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkz7q\" (UniqueName: \"kubernetes.io/projected/9f993f43-1925-403a-9fd5-2d7e9e1159f9-kube-api-access-bkz7q\") pod \"console-dc87ddb86-5gm4v\" (UID: \"9f993f43-1925-403a-9fd5-2d7e9e1159f9\") " pod="openshift-console/console-dc87ddb86-5gm4v" Jul 27 20:37:14.598806 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:14.598752 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f993f43-1925-403a-9fd5-2d7e9e1159f9-console-oauth-config\") pod \"console-dc87ddb86-5gm4v\" (UID: \"9f993f43-1925-403a-9fd5-2d7e9e1159f9\") " pod="openshift-console/console-dc87ddb86-5gm4v" Jul 27 20:37:14.598806 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:14.598783 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f993f43-1925-403a-9fd5-2d7e9e1159f9-console-config\") pod \"console-dc87ddb86-5gm4v\" (UID: \"9f993f43-1925-403a-9fd5-2d7e9e1159f9\") " pod="openshift-console/console-dc87ddb86-5gm4v" Jul 27 20:37:14.598806 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:14.598806 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f993f43-1925-403a-9fd5-2d7e9e1159f9-service-ca\") pod \"console-dc87ddb86-5gm4v\" (UID: \"9f993f43-1925-403a-9fd5-2d7e9e1159f9\") " pod="openshift-console/console-dc87ddb86-5gm4v" Jul 27 20:37:14.699404 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:14.699371 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f993f43-1925-403a-9fd5-2d7e9e1159f9-trusted-ca-bundle\") pod \"console-dc87ddb86-5gm4v\" (UID: \"9f993f43-1925-403a-9fd5-2d7e9e1159f9\") " pod="openshift-console/console-dc87ddb86-5gm4v" Jul 27 20:37:14.699960 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:14.699426 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f993f43-1925-403a-9fd5-2d7e9e1159f9-console-serving-cert\") pod \"console-dc87ddb86-5gm4v\" (UID: \"9f993f43-1925-403a-9fd5-2d7e9e1159f9\") " pod="openshift-console/console-dc87ddb86-5gm4v" Jul 27 20:37:14.699960 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:14.699548 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f993f43-1925-403a-9fd5-2d7e9e1159f9-oauth-serving-cert\") pod \"console-dc87ddb86-5gm4v\" (UID: \"9f993f43-1925-403a-9fd5-2d7e9e1159f9\") " pod="openshift-console/console-dc87ddb86-5gm4v" Jul 27 20:37:14.699960 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:14.699692 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkz7q\" (UniqueName: \"kubernetes.io/projected/9f993f43-1925-403a-9fd5-2d7e9e1159f9-kube-api-access-bkz7q\") pod \"console-dc87ddb86-5gm4v\" (UID: \"9f993f43-1925-403a-9fd5-2d7e9e1159f9\") " pod="openshift-console/console-dc87ddb86-5gm4v" Jul 27 20:37:14.699960 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:14.699768 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f993f43-1925-403a-9fd5-2d7e9e1159f9-console-oauth-config\") pod \"console-dc87ddb86-5gm4v\" (UID: \"9f993f43-1925-403a-9fd5-2d7e9e1159f9\") " pod="openshift-console/console-dc87ddb86-5gm4v" Jul 27 20:37:14.699960 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:14.699824 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f993f43-1925-403a-9fd5-2d7e9e1159f9-console-config\") pod \"console-dc87ddb86-5gm4v\" (UID: \"9f993f43-1925-403a-9fd5-2d7e9e1159f9\") " pod="openshift-console/console-dc87ddb86-5gm4v" Jul 27 20:37:14.699960 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:14.699863 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f993f43-1925-403a-9fd5-2d7e9e1159f9-service-ca\") pod \"console-dc87ddb86-5gm4v\" (UID: \"9f993f43-1925-403a-9fd5-2d7e9e1159f9\") " pod="openshift-console/console-dc87ddb86-5gm4v" Jul 27 20:37:14.700338 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:14.700308 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f993f43-1925-403a-9fd5-2d7e9e1159f9-oauth-serving-cert\") pod \"console-dc87ddb86-5gm4v\" (UID: \"9f993f43-1925-403a-9fd5-2d7e9e1159f9\") " pod="openshift-console/console-dc87ddb86-5gm4v" Jul 27 20:37:14.700518 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:14.700419 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f993f43-1925-403a-9fd5-2d7e9e1159f9-trusted-ca-bundle\") pod \"console-dc87ddb86-5gm4v\" (UID: \"9f993f43-1925-403a-9fd5-2d7e9e1159f9\") " pod="openshift-console/console-dc87ddb86-5gm4v" Jul 27 20:37:14.700806 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:14.700673 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f993f43-1925-403a-9fd5-2d7e9e1159f9-console-config\") pod \"console-dc87ddb86-5gm4v\" (UID: \"9f993f43-1925-403a-9fd5-2d7e9e1159f9\") " pod="openshift-console/console-dc87ddb86-5gm4v" Jul 27 20:37:14.700806 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:14.700737 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f993f43-1925-403a-9fd5-2d7e9e1159f9-service-ca\") pod \"console-dc87ddb86-5gm4v\" (UID: \"9f993f43-1925-403a-9fd5-2d7e9e1159f9\") " pod="openshift-console/console-dc87ddb86-5gm4v" Jul 27 20:37:14.702235 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:14.702212 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f993f43-1925-403a-9fd5-2d7e9e1159f9-console-serving-cert\") pod \"console-dc87ddb86-5gm4v\" (UID: \"9f993f43-1925-403a-9fd5-2d7e9e1159f9\") " pod="openshift-console/console-dc87ddb86-5gm4v" Jul 27 20:37:14.702374 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:14.702357 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f993f43-1925-403a-9fd5-2d7e9e1159f9-console-oauth-config\") pod \"console-dc87ddb86-5gm4v\" (UID: \"9f993f43-1925-403a-9fd5-2d7e9e1159f9\") " pod="openshift-console/console-dc87ddb86-5gm4v" Jul 27 20:37:14.709066 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:14.709040 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkz7q\" (UniqueName: \"kubernetes.io/projected/9f993f43-1925-403a-9fd5-2d7e9e1159f9-kube-api-access-bkz7q\") pod \"console-dc87ddb86-5gm4v\" (UID: \"9f993f43-1925-403a-9fd5-2d7e9e1159f9\") " pod="openshift-console/console-dc87ddb86-5gm4v" Jul 27 20:37:14.863132 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:14.863056 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dc87ddb86-5gm4v" Jul 27 20:37:14.994009 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:14.993983 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dc87ddb86-5gm4v"] Jul 27 20:37:14.996359 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:37:14.996333 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f993f43_1925_403a_9fd5_2d7e9e1159f9.slice/crio-b64204e9696ca1c833fb352d7f34a1ec253b0b3865c73678e34fdefb5ffd7739 WatchSource:0}: Error finding container b64204e9696ca1c833fb352d7f34a1ec253b0b3865c73678e34fdefb5ffd7739: Status 404 returned error can't find the container with id b64204e9696ca1c833fb352d7f34a1ec253b0b3865c73678e34fdefb5ffd7739 Jul 27 20:37:15.633514 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:15.633476 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dc87ddb86-5gm4v" event={"ID":"9f993f43-1925-403a-9fd5-2d7e9e1159f9","Type":"ContainerStarted","Data":"5182ba1a6767038cfebe691abfa46037b6090ba91507bfd05e3e682d0b8e8847"} Jul 27 20:37:15.633708 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:15.633530 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dc87ddb86-5gm4v" event={"ID":"9f993f43-1925-403a-9fd5-2d7e9e1159f9","Type":"ContainerStarted","Data":"b64204e9696ca1c833fb352d7f34a1ec253b0b3865c73678e34fdefb5ffd7739"} Jul 27 20:37:15.652450 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:15.652394 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-dc87ddb86-5gm4v" podStartSLOduration=1.652375541 podStartE2EDuration="1.652375541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:37:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:37:15.651573242 +0000 UTC m=+165.387553531" watchObservedRunningTime="2026-07-27 20:37:15.652375541 +0000 UTC m=+165.388355879" Jul 27 20:37:24.863269 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:24.863173 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-dc87ddb86-5gm4v" Jul 27 20:37:24.863269 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:24.863243 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-dc87ddb86-5gm4v" Jul 27 20:37:24.868037 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:24.868012 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-dc87ddb86-5gm4v" Jul 27 20:37:25.678417 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:37:25.678372 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-dc87ddb86-5gm4v" Jul 27 20:38:31.354881 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:38:31.354851 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c484c9c79-ttn8p"] Jul 27 20:38:31.358063 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:38:31.358035 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c484c9c79-ttn8p" Jul 27 20:38:31.372307 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:38:31.372277 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c484c9c79-ttn8p"] Jul 27 20:38:31.427401 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:38:31.427359 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c3952d4-3b16-4463-83e6-a59682647929-console-serving-cert\") pod \"console-5c484c9c79-ttn8p\" (UID: \"4c3952d4-3b16-4463-83e6-a59682647929\") " pod="openshift-console/console-5c484c9c79-ttn8p" Jul 27 20:38:31.427401 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:38:31.427398 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c3952d4-3b16-4463-83e6-a59682647929-service-ca\") pod \"console-5c484c9c79-ttn8p\" (UID: \"4c3952d4-3b16-4463-83e6-a59682647929\") " pod="openshift-console/console-5c484c9c79-ttn8p" Jul 27 20:38:31.427669 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:38:31.427425 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c3952d4-3b16-4463-83e6-a59682647929-trusted-ca-bundle\") pod \"console-5c484c9c79-ttn8p\" (UID: \"4c3952d4-3b16-4463-83e6-a59682647929\") " pod="openshift-console/console-5c484c9c79-ttn8p" Jul 27 20:38:31.427669 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:38:31.427500 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd5rq\" (UniqueName: \"kubernetes.io/projected/4c3952d4-3b16-4463-83e6-a59682647929-kube-api-access-cd5rq\") pod \"console-5c484c9c79-ttn8p\" (UID: \"4c3952d4-3b16-4463-83e6-a59682647929\") " pod="openshift-console/console-5c484c9c79-ttn8p" Jul 27 20:38:31.427669 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:38:31.427550 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c3952d4-3b16-4463-83e6-a59682647929-console-config\") pod \"console-5c484c9c79-ttn8p\" (UID: \"4c3952d4-3b16-4463-83e6-a59682647929\") " pod="openshift-console/console-5c484c9c79-ttn8p" Jul 27 20:38:31.427669 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:38:31.427613 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c3952d4-3b16-4463-83e6-a59682647929-oauth-serving-cert\") pod \"console-5c484c9c79-ttn8p\" (UID: \"4c3952d4-3b16-4463-83e6-a59682647929\") " pod="openshift-console/console-5c484c9c79-ttn8p" Jul 27 20:38:31.427669 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:38:31.427648 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c3952d4-3b16-4463-83e6-a59682647929-console-oauth-config\") pod \"console-5c484c9c79-ttn8p\" (UID: \"4c3952d4-3b16-4463-83e6-a59682647929\") " pod="openshift-console/console-5c484c9c79-ttn8p" Jul 27 20:38:31.528534 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:38:31.528490 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c3952d4-3b16-4463-83e6-a59682647929-console-serving-cert\") pod \"console-5c484c9c79-ttn8p\" (UID: \"4c3952d4-3b16-4463-83e6-a59682647929\") " pod="openshift-console/console-5c484c9c79-ttn8p" Jul 27 20:38:31.528534 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:38:31.528530 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c3952d4-3b16-4463-83e6-a59682647929-service-ca\") pod \"console-5c484c9c79-ttn8p\" (UID: \"4c3952d4-3b16-4463-83e6-a59682647929\") " pod="openshift-console/console-5c484c9c79-ttn8p" Jul 27 20:38:31.528792 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:38:31.528559 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c3952d4-3b16-4463-83e6-a59682647929-trusted-ca-bundle\") pod \"console-5c484c9c79-ttn8p\" (UID: \"4c3952d4-3b16-4463-83e6-a59682647929\") " pod="openshift-console/console-5c484c9c79-ttn8p" Jul 27 20:38:31.528792 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:38:31.528584 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd5rq\" (UniqueName: \"kubernetes.io/projected/4c3952d4-3b16-4463-83e6-a59682647929-kube-api-access-cd5rq\") pod \"console-5c484c9c79-ttn8p\" (UID: \"4c3952d4-3b16-4463-83e6-a59682647929\") " pod="openshift-console/console-5c484c9c79-ttn8p" Jul 27 20:38:31.528792 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:38:31.528617 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c3952d4-3b16-4463-83e6-a59682647929-console-config\") pod \"console-5c484c9c79-ttn8p\" (UID: \"4c3952d4-3b16-4463-83e6-a59682647929\") " pod="openshift-console/console-5c484c9c79-ttn8p" Jul 27 20:38:31.528792 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:38:31.528644 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c3952d4-3b16-4463-83e6-a59682647929-oauth-serving-cert\") pod \"console-5c484c9c79-ttn8p\" (UID: \"4c3952d4-3b16-4463-83e6-a59682647929\") " pod="openshift-console/console-5c484c9c79-ttn8p" Jul 27 20:38:31.528958 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:38:31.528795 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c3952d4-3b16-4463-83e6-a59682647929-console-oauth-config\") pod \"console-5c484c9c79-ttn8p\" (UID: \"4c3952d4-3b16-4463-83e6-a59682647929\") " pod="openshift-console/console-5c484c9c79-ttn8p" Jul 27 20:38:31.529338 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:38:31.529311 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c3952d4-3b16-4463-83e6-a59682647929-service-ca\") pod \"console-5c484c9c79-ttn8p\" (UID: \"4c3952d4-3b16-4463-83e6-a59682647929\") " pod="openshift-console/console-5c484c9c79-ttn8p" Jul 27 20:38:31.529476 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:38:31.529460 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c3952d4-3b16-4463-83e6-a59682647929-oauth-serving-cert\") pod \"console-5c484c9c79-ttn8p\" (UID: \"4c3952d4-3b16-4463-83e6-a59682647929\") " pod="openshift-console/console-5c484c9c79-ttn8p" Jul 27 20:38:31.529822 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:38:31.529799 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c3952d4-3b16-4463-83e6-a59682647929-console-config\") pod \"console-5c484c9c79-ttn8p\" (UID: \"4c3952d4-3b16-4463-83e6-a59682647929\") " pod="openshift-console/console-5c484c9c79-ttn8p" Jul 27 20:38:31.529874 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:38:31.529811 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c3952d4-3b16-4463-83e6-a59682647929-trusted-ca-bundle\") pod \"console-5c484c9c79-ttn8p\" (UID: \"4c3952d4-3b16-4463-83e6-a59682647929\") " pod="openshift-console/console-5c484c9c79-ttn8p" Jul 27 20:38:31.531387 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:38:31.531369 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c3952d4-3b16-4463-83e6-a59682647929-console-serving-cert\") pod \"console-5c484c9c79-ttn8p\" (UID: \"4c3952d4-3b16-4463-83e6-a59682647929\") " pod="openshift-console/console-5c484c9c79-ttn8p" Jul 27 20:38:31.531481 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:38:31.531399 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c3952d4-3b16-4463-83e6-a59682647929-console-oauth-config\") pod \"console-5c484c9c79-ttn8p\" (UID: \"4c3952d4-3b16-4463-83e6-a59682647929\") " pod="openshift-console/console-5c484c9c79-ttn8p" Jul 27 20:38:31.538238 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:38:31.538209 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd5rq\" (UniqueName: \"kubernetes.io/projected/4c3952d4-3b16-4463-83e6-a59682647929-kube-api-access-cd5rq\") pod \"console-5c484c9c79-ttn8p\" (UID: \"4c3952d4-3b16-4463-83e6-a59682647929\") " pod="openshift-console/console-5c484c9c79-ttn8p" Jul 27 20:38:31.667428 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:38:31.667331 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c484c9c79-ttn8p" Jul 27 20:38:31.805871 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:38:31.805842 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c484c9c79-ttn8p"] Jul 27 20:38:31.808305 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:38:31.808277 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c3952d4_3b16_4463_83e6_a59682647929.slice/crio-7da1e6a8e5f5dc7ec0d9dc6ae87a7a692e50e85af85d7c83c4d868d999fddb28 WatchSource:0}: Error finding container 7da1e6a8e5f5dc7ec0d9dc6ae87a7a692e50e85af85d7c83c4d868d999fddb28: Status 404 returned error can't find the container with id 7da1e6a8e5f5dc7ec0d9dc6ae87a7a692e50e85af85d7c83c4d868d999fddb28 Jul 27 20:38:31.933935 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:38:31.933854 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c484c9c79-ttn8p" event={"ID":"4c3952d4-3b16-4463-83e6-a59682647929","Type":"ContainerStarted","Data":"cb83d54ee8409d606d12b15913ba24d88c28299dd31593db1332cfb041182f80"} Jul 27 20:38:31.933935 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:38:31.933907 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c484c9c79-ttn8p" event={"ID":"4c3952d4-3b16-4463-83e6-a59682647929","Type":"ContainerStarted","Data":"7da1e6a8e5f5dc7ec0d9dc6ae87a7a692e50e85af85d7c83c4d868d999fddb28"} Jul 27 20:38:31.953326 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:38:31.953260 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c484c9c79-ttn8p" podStartSLOduration=0.953240156 podStartE2EDuration="953.240156ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:38:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:38:31.952858094 +0000 UTC m=+241.688838382" watchObservedRunningTime="2026-07-27 20:38:31.953240156 +0000 UTC m=+241.689220444" Jul 27 20:38:41.668416 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:38:41.668364 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c484c9c79-ttn8p" Jul 27 20:38:41.668947 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:38:41.668443 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c484c9c79-ttn8p" Jul 27 20:38:41.673119 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:38:41.673092 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c484c9c79-ttn8p" Jul 27 20:38:41.977422 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:38:41.977397 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c484c9c79-ttn8p" Jul 27 20:38:42.026572 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:38:42.026537 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dc87ddb86-5gm4v"] Jul 27 20:39:07.050632 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:07.050499 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-dc87ddb86-5gm4v" podUID="9f993f43-1925-403a-9fd5-2d7e9e1159f9" containerName="console" containerID="cri-o://5182ba1a6767038cfebe691abfa46037b6090ba91507bfd05e3e682d0b8e8847" gracePeriod=15 Jul 27 20:39:07.299026 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:07.299001 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dc87ddb86-5gm4v" Jul 27 20:39:07.329880 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:07.329791 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f993f43-1925-403a-9fd5-2d7e9e1159f9-console-serving-cert\") pod \"9f993f43-1925-403a-9fd5-2d7e9e1159f9\" (UID: \"9f993f43-1925-403a-9fd5-2d7e9e1159f9\") " Jul 27 20:39:07.329880 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:07.329856 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkz7q\" (UniqueName: \"kubernetes.io/projected/9f993f43-1925-403a-9fd5-2d7e9e1159f9-kube-api-access-bkz7q\") pod \"9f993f43-1925-403a-9fd5-2d7e9e1159f9\" (UID: \"9f993f43-1925-403a-9fd5-2d7e9e1159f9\") " Jul 27 20:39:07.330100 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:07.329887 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f993f43-1925-403a-9fd5-2d7e9e1159f9-oauth-serving-cert\") pod \"9f993f43-1925-403a-9fd5-2d7e9e1159f9\" (UID: \"9f993f43-1925-403a-9fd5-2d7e9e1159f9\") " Jul 27 20:39:07.330100 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:07.329992 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f993f43-1925-403a-9fd5-2d7e9e1159f9-console-oauth-config\") pod \"9f993f43-1925-403a-9fd5-2d7e9e1159f9\" (UID: \"9f993f43-1925-403a-9fd5-2d7e9e1159f9\") " Jul 27 20:39:07.330100 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:07.330039 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f993f43-1925-403a-9fd5-2d7e9e1159f9-console-config\") pod \"9f993f43-1925-403a-9fd5-2d7e9e1159f9\" (UID: \"9f993f43-1925-403a-9fd5-2d7e9e1159f9\") " Jul 27 20:39:07.330100 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:07.330068 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f993f43-1925-403a-9fd5-2d7e9e1159f9-service-ca\") pod \"9f993f43-1925-403a-9fd5-2d7e9e1159f9\" (UID: \"9f993f43-1925-403a-9fd5-2d7e9e1159f9\") " Jul 27 20:39:07.330303 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:07.330120 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f993f43-1925-403a-9fd5-2d7e9e1159f9-trusted-ca-bundle\") pod \"9f993f43-1925-403a-9fd5-2d7e9e1159f9\" (UID: \"9f993f43-1925-403a-9fd5-2d7e9e1159f9\") " Jul 27 20:39:07.330405 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:07.330371 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f993f43-1925-403a-9fd5-2d7e9e1159f9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9f993f43-1925-403a-9fd5-2d7e9e1159f9" (UID: "9f993f43-1925-403a-9fd5-2d7e9e1159f9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:39:07.330557 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:07.330523 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f993f43-1925-403a-9fd5-2d7e9e1159f9-console-config" (OuterVolumeSpecName: "console-config") pod "9f993f43-1925-403a-9fd5-2d7e9e1159f9" (UID: "9f993f43-1925-403a-9fd5-2d7e9e1159f9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:39:07.330678 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:07.330498 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f993f43-1925-403a-9fd5-2d7e9e1159f9-service-ca" (OuterVolumeSpecName: "service-ca") pod "9f993f43-1925-403a-9fd5-2d7e9e1159f9" (UID: "9f993f43-1925-403a-9fd5-2d7e9e1159f9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:39:07.330678 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:07.330653 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f993f43-1925-403a-9fd5-2d7e9e1159f9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9f993f43-1925-403a-9fd5-2d7e9e1159f9" (UID: "9f993f43-1925-403a-9fd5-2d7e9e1159f9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:39:07.332289 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:07.332258 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f993f43-1925-403a-9fd5-2d7e9e1159f9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9f993f43-1925-403a-9fd5-2d7e9e1159f9" (UID: "9f993f43-1925-403a-9fd5-2d7e9e1159f9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 20:39:07.332380 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:07.332294 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f993f43-1925-403a-9fd5-2d7e9e1159f9-kube-api-access-bkz7q" (OuterVolumeSpecName: "kube-api-access-bkz7q") pod "9f993f43-1925-403a-9fd5-2d7e9e1159f9" (UID: "9f993f43-1925-403a-9fd5-2d7e9e1159f9"). InnerVolumeSpecName "kube-api-access-bkz7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:39:07.332484 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:07.332469 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f993f43-1925-403a-9fd5-2d7e9e1159f9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9f993f43-1925-403a-9fd5-2d7e9e1159f9" (UID: "9f993f43-1925-403a-9fd5-2d7e9e1159f9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 20:39:07.430997 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:07.430938 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f993f43-1925-403a-9fd5-2d7e9e1159f9-trusted-ca-bundle\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:39:07.430997 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:07.430988 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f993f43-1925-403a-9fd5-2d7e9e1159f9-console-serving-cert\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:39:07.430997 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:07.431002 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkz7q\" (UniqueName: \"kubernetes.io/projected/9f993f43-1925-403a-9fd5-2d7e9e1159f9-kube-api-access-bkz7q\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:39:07.431249 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:07.431017 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f993f43-1925-403a-9fd5-2d7e9e1159f9-oauth-serving-cert\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:39:07.431249 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:07.431028 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f993f43-1925-403a-9fd5-2d7e9e1159f9-console-oauth-config\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:39:07.431249 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:07.431041 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f993f43-1925-403a-9fd5-2d7e9e1159f9-console-config\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:39:07.431249 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:07.431054 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f993f43-1925-403a-9fd5-2d7e9e1159f9-service-ca\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:39:08.072935 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:08.072902 2570 generic.go:356] "Generic (PLEG): container finished" podID="9f993f43-1925-403a-9fd5-2d7e9e1159f9" containerID="5182ba1a6767038cfebe691abfa46037b6090ba91507bfd05e3e682d0b8e8847" exitCode=0 Jul 27 20:39:08.073365 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:08.072978 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dc87ddb86-5gm4v" Jul 27 20:39:08.073365 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:08.072979 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dc87ddb86-5gm4v" event={"ID":"9f993f43-1925-403a-9fd5-2d7e9e1159f9","Type":"ContainerDied","Data":"5182ba1a6767038cfebe691abfa46037b6090ba91507bfd05e3e682d0b8e8847"} Jul 27 20:39:08.073365 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:08.073047 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dc87ddb86-5gm4v" event={"ID":"9f993f43-1925-403a-9fd5-2d7e9e1159f9","Type":"ContainerDied","Data":"b64204e9696ca1c833fb352d7f34a1ec253b0b3865c73678e34fdefb5ffd7739"} Jul 27 20:39:08.073365 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:08.073075 2570 scope.go:117] "RemoveContainer" containerID="5182ba1a6767038cfebe691abfa46037b6090ba91507bfd05e3e682d0b8e8847" Jul 27 20:39:08.082517 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:08.082498 2570 scope.go:117] "RemoveContainer" containerID="5182ba1a6767038cfebe691abfa46037b6090ba91507bfd05e3e682d0b8e8847" Jul 27 20:39:08.082865 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:39:08.082845 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5182ba1a6767038cfebe691abfa46037b6090ba91507bfd05e3e682d0b8e8847\": container with ID starting with 5182ba1a6767038cfebe691abfa46037b6090ba91507bfd05e3e682d0b8e8847 not found: ID does not exist" containerID="5182ba1a6767038cfebe691abfa46037b6090ba91507bfd05e3e682d0b8e8847" Jul 27 20:39:08.082922 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:08.082875 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5182ba1a6767038cfebe691abfa46037b6090ba91507bfd05e3e682d0b8e8847"} err="failed to get container status \"5182ba1a6767038cfebe691abfa46037b6090ba91507bfd05e3e682d0b8e8847\": rpc error: code = NotFound desc = could not find container \"5182ba1a6767038cfebe691abfa46037b6090ba91507bfd05e3e682d0b8e8847\": container with ID starting with 5182ba1a6767038cfebe691abfa46037b6090ba91507bfd05e3e682d0b8e8847 not found: ID does not exist" Jul 27 20:39:08.094015 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:08.093979 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dc87ddb86-5gm4v"] Jul 27 20:39:08.097561 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:08.097532 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-dc87ddb86-5gm4v"] Jul 27 20:39:08.852661 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:08.852624 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f993f43-1925-403a-9fd5-2d7e9e1159f9" path="/var/lib/kubelet/pods/9f993f43-1925-403a-9fd5-2d7e9e1159f9/volumes" Jul 27 20:39:28.414873 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:28.414835 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbhxd"] Jul 27 20:39:28.415465 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:28.415180 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f993f43-1925-403a-9fd5-2d7e9e1159f9" containerName="console" Jul 27 20:39:28.415465 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:28.415191 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f993f43-1925-403a-9fd5-2d7e9e1159f9" containerName="console" Jul 27 20:39:28.415465 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:28.415248 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f993f43-1925-403a-9fd5-2d7e9e1159f9" containerName="console" Jul 27 20:39:28.418157 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:28.418139 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbhxd" Jul 27 20:39:28.420862 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:28.420841 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ntvl4\"" Jul 27 20:39:28.420976 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:28.420844 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 27 20:39:28.421566 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:28.421545 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 27 20:39:28.426433 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:28.426413 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbhxd"] Jul 27 20:39:28.519779 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:28.519739 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69542336-b8e8-4149-ac54-162c9e9869ee-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbhxd\" (UID: \"69542336-b8e8-4149-ac54-162c9e9869ee\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbhxd" Jul 27 20:39:28.519779 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:28.519789 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps77g\" (UniqueName: \"kubernetes.io/projected/69542336-b8e8-4149-ac54-162c9e9869ee-kube-api-access-ps77g\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbhxd\" (UID: \"69542336-b8e8-4149-ac54-162c9e9869ee\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbhxd" Jul 27 20:39:28.520020 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:28.519833 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69542336-b8e8-4149-ac54-162c9e9869ee-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbhxd\" (UID: \"69542336-b8e8-4149-ac54-162c9e9869ee\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbhxd" Jul 27 20:39:28.620531 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:28.620476 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69542336-b8e8-4149-ac54-162c9e9869ee-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbhxd\" (UID: \"69542336-b8e8-4149-ac54-162c9e9869ee\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbhxd" Jul 27 20:39:28.620776 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:28.620552 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps77g\" (UniqueName: \"kubernetes.io/projected/69542336-b8e8-4149-ac54-162c9e9869ee-kube-api-access-ps77g\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbhxd\" (UID: \"69542336-b8e8-4149-ac54-162c9e9869ee\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbhxd" Jul 27 20:39:28.620776 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:28.620628 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69542336-b8e8-4149-ac54-162c9e9869ee-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbhxd\" (UID: \"69542336-b8e8-4149-ac54-162c9e9869ee\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbhxd" Jul 27 20:39:28.620923 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:28.620904 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69542336-b8e8-4149-ac54-162c9e9869ee-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbhxd\" (UID: \"69542336-b8e8-4149-ac54-162c9e9869ee\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbhxd" Jul 27 20:39:28.620975 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:28.620938 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69542336-b8e8-4149-ac54-162c9e9869ee-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbhxd\" (UID: \"69542336-b8e8-4149-ac54-162c9e9869ee\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbhxd" Jul 27 20:39:28.629581 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:28.629559 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps77g\" (UniqueName: \"kubernetes.io/projected/69542336-b8e8-4149-ac54-162c9e9869ee-kube-api-access-ps77g\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbhxd\" (UID: \"69542336-b8e8-4149-ac54-162c9e9869ee\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbhxd" Jul 27 20:39:28.728372 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:28.728331 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbhxd" Jul 27 20:39:28.862296 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:28.862272 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbhxd"] Jul 27 20:39:28.864771 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:39:28.864735 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69542336_b8e8_4149_ac54_162c9e9869ee.slice/crio-6abbff4baa8a25da0e1e0f17dba0e7f68fa2dad12741850b6f956c3a9b4010e6 WatchSource:0}: Error finding container 6abbff4baa8a25da0e1e0f17dba0e7f68fa2dad12741850b6f956c3a9b4010e6: Status 404 returned error can't find the container with id 6abbff4baa8a25da0e1e0f17dba0e7f68fa2dad12741850b6f956c3a9b4010e6 Jul 27 20:39:29.154134 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:29.154024 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbhxd" event={"ID":"69542336-b8e8-4149-ac54-162c9e9869ee","Type":"ContainerStarted","Data":"6abbff4baa8a25da0e1e0f17dba0e7f68fa2dad12741850b6f956c3a9b4010e6"} Jul 27 20:39:30.672694 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:30.672667 2570 kubelet.go:1648] "Image garbage collection succeeded" Jul 27 20:39:36.184300 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:36.184260 2570 generic.go:356] "Generic (PLEG): container finished" podID="69542336-b8e8-4149-ac54-162c9e9869ee" containerID="2443bc600cc6642a42fb214c86a8d2b4c2d1cbe6fa4c67d053e4acbd71375cc9" exitCode=0 Jul 27 20:39:36.184745 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:36.184343 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbhxd" event={"ID":"69542336-b8e8-4149-ac54-162c9e9869ee","Type":"ContainerDied","Data":"2443bc600cc6642a42fb214c86a8d2b4c2d1cbe6fa4c67d053e4acbd71375cc9"} Jul 27 20:39:36.185399 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:36.185378 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 20:39:39.197117 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:39.197076 2570 generic.go:356] "Generic (PLEG): container finished" podID="69542336-b8e8-4149-ac54-162c9e9869ee" containerID="d5282672a35f556e2363171f62566c7dbb150a5f00db23e38c0c8e512e2f92f3" exitCode=0 Jul 27 20:39:39.197542 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:39.197168 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbhxd" event={"ID":"69542336-b8e8-4149-ac54-162c9e9869ee","Type":"ContainerDied","Data":"d5282672a35f556e2363171f62566c7dbb150a5f00db23e38c0c8e512e2f92f3"} Jul 27 20:39:47.234361 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:47.234322 2570 generic.go:356] "Generic (PLEG): container finished" podID="69542336-b8e8-4149-ac54-162c9e9869ee" containerID="866662badb0eb2d891e6dd0e712219aba4b3312224ba6cf69f2b67158f70887e" exitCode=0 Jul 27 20:39:47.234873 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:47.234403 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbhxd" event={"ID":"69542336-b8e8-4149-ac54-162c9e9869ee","Type":"ContainerDied","Data":"866662badb0eb2d891e6dd0e712219aba4b3312224ba6cf69f2b67158f70887e"} Jul 27 20:39:48.366556 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:48.366531 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbhxd" Jul 27 20:39:48.476391 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:48.476353 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69542336-b8e8-4149-ac54-162c9e9869ee-util\") pod \"69542336-b8e8-4149-ac54-162c9e9869ee\" (UID: \"69542336-b8e8-4149-ac54-162c9e9869ee\") " Jul 27 20:39:48.476561 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:48.476427 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ps77g\" (UniqueName: \"kubernetes.io/projected/69542336-b8e8-4149-ac54-162c9e9869ee-kube-api-access-ps77g\") pod \"69542336-b8e8-4149-ac54-162c9e9869ee\" (UID: \"69542336-b8e8-4149-ac54-162c9e9869ee\") " Jul 27 20:39:48.476561 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:48.476447 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69542336-b8e8-4149-ac54-162c9e9869ee-bundle\") pod \"69542336-b8e8-4149-ac54-162c9e9869ee\" (UID: \"69542336-b8e8-4149-ac54-162c9e9869ee\") " Jul 27 20:39:48.477208 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:48.477173 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69542336-b8e8-4149-ac54-162c9e9869ee-bundle" (OuterVolumeSpecName: "bundle") pod "69542336-b8e8-4149-ac54-162c9e9869ee" (UID: "69542336-b8e8-4149-ac54-162c9e9869ee"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 20:39:48.478963 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:48.478924 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69542336-b8e8-4149-ac54-162c9e9869ee-kube-api-access-ps77g" (OuterVolumeSpecName: "kube-api-access-ps77g") pod "69542336-b8e8-4149-ac54-162c9e9869ee" (UID: "69542336-b8e8-4149-ac54-162c9e9869ee"). InnerVolumeSpecName "kube-api-access-ps77g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:39:48.480557 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:48.480532 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69542336-b8e8-4149-ac54-162c9e9869ee-util" (OuterVolumeSpecName: "util") pod "69542336-b8e8-4149-ac54-162c9e9869ee" (UID: "69542336-b8e8-4149-ac54-162c9e9869ee"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 20:39:48.577200 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:48.577108 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps77g\" (UniqueName: \"kubernetes.io/projected/69542336-b8e8-4149-ac54-162c9e9869ee-kube-api-access-ps77g\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:39:48.577200 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:48.577141 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69542336-b8e8-4149-ac54-162c9e9869ee-bundle\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:39:48.577200 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:48.577155 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69542336-b8e8-4149-ac54-162c9e9869ee-util\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:39:49.244337 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:49.244306 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbhxd" event={"ID":"69542336-b8e8-4149-ac54-162c9e9869ee","Type":"ContainerDied","Data":"6abbff4baa8a25da0e1e0f17dba0e7f68fa2dad12741850b6f956c3a9b4010e6"} Jul 27 20:39:49.244504 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:49.244352 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6abbff4baa8a25da0e1e0f17dba0e7f68fa2dad12741850b6f956c3a9b4010e6" Jul 27 20:39:49.244504 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:49.244353 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hbhxd" Jul 27 20:39:56.441575 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:56.441540 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sl62k"] Jul 27 20:39:56.442079 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:56.441859 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69542336-b8e8-4149-ac54-162c9e9869ee" containerName="extract" Jul 27 20:39:56.442079 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:56.441870 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="69542336-b8e8-4149-ac54-162c9e9869ee" containerName="extract" Jul 27 20:39:56.442079 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:56.441885 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69542336-b8e8-4149-ac54-162c9e9869ee" containerName="pull" Jul 27 20:39:56.442079 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:56.441890 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="69542336-b8e8-4149-ac54-162c9e9869ee" containerName="pull" Jul 27 20:39:56.442079 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:56.441899 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69542336-b8e8-4149-ac54-162c9e9869ee" containerName="util" Jul 27 20:39:56.442079 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:56.441905 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="69542336-b8e8-4149-ac54-162c9e9869ee" containerName="util" Jul 27 20:39:56.442079 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:56.441963 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="69542336-b8e8-4149-ac54-162c9e9869ee" containerName="extract" Jul 27 20:39:56.444168 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:56.444148 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sl62k" Jul 27 20:39:56.446767 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:56.446751 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 27 20:39:56.447044 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:56.447029 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 27 20:39:56.447329 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:56.447313 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-5rrv7\"" Jul 27 20:39:56.458440 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:56.458414 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sl62k"] Jul 27 20:39:56.545224 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:56.545175 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3115bba-f957-4548-b952-14af55741d57-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-sl62k\" (UID: \"b3115bba-f957-4548-b952-14af55741d57\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sl62k" Jul 27 20:39:56.545428 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:56.545296 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x27g\" (UniqueName: \"kubernetes.io/projected/b3115bba-f957-4548-b952-14af55741d57-kube-api-access-9x27g\") pod \"cert-manager-operator-controller-manager-55c444bf86-sl62k\" (UID: \"b3115bba-f957-4548-b952-14af55741d57\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sl62k" Jul 27 20:39:56.646051 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:56.646004 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3115bba-f957-4548-b952-14af55741d57-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-sl62k\" (UID: \"b3115bba-f957-4548-b952-14af55741d57\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sl62k" Jul 27 20:39:56.646251 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:56.646098 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x27g\" (UniqueName: \"kubernetes.io/projected/b3115bba-f957-4548-b952-14af55741d57-kube-api-access-9x27g\") pod \"cert-manager-operator-controller-manager-55c444bf86-sl62k\" (UID: \"b3115bba-f957-4548-b952-14af55741d57\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sl62k" Jul 27 20:39:56.646399 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:56.646373 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3115bba-f957-4548-b952-14af55741d57-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-sl62k\" (UID: \"b3115bba-f957-4548-b952-14af55741d57\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sl62k" Jul 27 20:39:56.655869 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:56.655842 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x27g\" (UniqueName: \"kubernetes.io/projected/b3115bba-f957-4548-b952-14af55741d57-kube-api-access-9x27g\") pod \"cert-manager-operator-controller-manager-55c444bf86-sl62k\" (UID: \"b3115bba-f957-4548-b952-14af55741d57\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sl62k" Jul 27 20:39:56.753644 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:56.753588 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sl62k" Jul 27 20:39:56.893044 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:56.893020 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sl62k"] Jul 27 20:39:56.895030 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:39:56.895002 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3115bba_f957_4548_b952_14af55741d57.slice/crio-0ac5f889f00f94497f5f7eac1ae3cf9cd0f4c16a5a7d6bc430ace6283c43b03e WatchSource:0}: Error finding container 0ac5f889f00f94497f5f7eac1ae3cf9cd0f4c16a5a7d6bc430ace6283c43b03e: Status 404 returned error can't find the container with id 0ac5f889f00f94497f5f7eac1ae3cf9cd0f4c16a5a7d6bc430ace6283c43b03e Jul 27 20:39:57.278045 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:57.278007 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sl62k" event={"ID":"b3115bba-f957-4548-b952-14af55741d57","Type":"ContainerStarted","Data":"0ac5f889f00f94497f5f7eac1ae3cf9cd0f4c16a5a7d6bc430ace6283c43b03e"} Jul 27 20:39:59.288861 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:59.288822 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sl62k" event={"ID":"b3115bba-f957-4548-b952-14af55741d57","Type":"ContainerStarted","Data":"bd7879da1591f02f3c4142fc456e6c177c754ffc258e322e71ace3bac503a0d2"} Jul 27 20:39:59.314591 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:59.314520 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sl62k" podStartSLOduration=1.646812057 podStartE2EDuration="3.314500684s" totalImagesPullingTime="1.667688627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:39:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:39:59.313654761 +0000 UTC m=+329.049635047" watchObservedRunningTime="2026-07-27 20:39:59.314500684 +0000 UTC m=+329.050480971" Jul 27 20:39:59.361729 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:59.361698 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-z95wq"] Jul 27 20:39:59.364180 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:59.364157 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-z95wq" Jul 27 20:39:59.366885 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:59.366868 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-d4djs\"" Jul 27 20:39:59.366990 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:59.366907 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 27 20:39:59.366990 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:59.366908 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 27 20:39:59.377049 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:59.377026 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-z95wq"] Jul 27 20:39:59.468839 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:59.468790 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8cfl\" (UniqueName: \"kubernetes.io/projected/3c9a25ad-a351-40d7-bd1c-8f8968064f7e-kube-api-access-k8cfl\") pod \"cert-manager-webhook-6486ffd997-z95wq\" (UID: \"3c9a25ad-a351-40d7-bd1c-8f8968064f7e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-z95wq" Jul 27 20:39:59.468839 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:59.468841 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3c9a25ad-a351-40d7-bd1c-8f8968064f7e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-z95wq\" (UID: \"3c9a25ad-a351-40d7-bd1c-8f8968064f7e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-z95wq" Jul 27 20:39:59.569640 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:59.569537 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8cfl\" (UniqueName: \"kubernetes.io/projected/3c9a25ad-a351-40d7-bd1c-8f8968064f7e-kube-api-access-k8cfl\") pod \"cert-manager-webhook-6486ffd997-z95wq\" (UID: \"3c9a25ad-a351-40d7-bd1c-8f8968064f7e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-z95wq" Jul 27 20:39:59.569640 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:59.569573 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3c9a25ad-a351-40d7-bd1c-8f8968064f7e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-z95wq\" (UID: \"3c9a25ad-a351-40d7-bd1c-8f8968064f7e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-z95wq" Jul 27 20:39:59.578120 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:59.578097 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8cfl\" (UniqueName: \"kubernetes.io/projected/3c9a25ad-a351-40d7-bd1c-8f8968064f7e-kube-api-access-k8cfl\") pod \"cert-manager-webhook-6486ffd997-z95wq\" (UID: \"3c9a25ad-a351-40d7-bd1c-8f8968064f7e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-z95wq" Jul 27 20:39:59.578120 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:59.578102 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3c9a25ad-a351-40d7-bd1c-8f8968064f7e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-z95wq\" (UID: \"3c9a25ad-a351-40d7-bd1c-8f8968064f7e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-z95wq" Jul 27 20:39:59.673407 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:59.673367 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-z95wq" Jul 27 20:39:59.821098 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:39:59.821063 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-z95wq"] Jul 27 20:39:59.824092 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:39:59.824063 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c9a25ad_a351_40d7_bd1c_8f8968064f7e.slice/crio-a534ec0793315eb2d0b79aa4b0d57d9a84eb855664cffd2eeefe36356d823a8a WatchSource:0}: Error finding container a534ec0793315eb2d0b79aa4b0d57d9a84eb855664cffd2eeefe36356d823a8a: Status 404 returned error can't find the container with id a534ec0793315eb2d0b79aa4b0d57d9a84eb855664cffd2eeefe36356d823a8a Jul 27 20:40:00.294066 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:00.294027 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-z95wq" event={"ID":"3c9a25ad-a351-40d7-bd1c-8f8968064f7e","Type":"ContainerStarted","Data":"a534ec0793315eb2d0b79aa4b0d57d9a84eb855664cffd2eeefe36356d823a8a"} Jul 27 20:40:03.310890 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:03.310846 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-z95wq" event={"ID":"3c9a25ad-a351-40d7-bd1c-8f8968064f7e","Type":"ContainerStarted","Data":"5d49530a111c5331a7865f06dae0cb87fd58f5467a05faf84faa487c5570c769"} Jul 27 20:40:03.311384 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:03.310911 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-z95wq" Jul 27 20:40:03.329119 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:03.329066 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-z95wq" podStartSLOduration=1.478260942 podStartE2EDuration="4.329051568s" totalImagesPullingTime="2.850790626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:39:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:40:03.327695875 +0000 UTC m=+333.063676162" watchObservedRunningTime="2026-07-27 20:40:03.329051568 +0000 UTC m=+333.065031855" Jul 27 20:40:09.317355 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:09.317320 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-z95wq" Jul 27 20:40:36.135020 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:36.134942 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lbx57"] Jul 27 20:40:36.137445 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:36.137422 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lbx57" Jul 27 20:40:36.140042 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:36.140019 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 27 20:40:36.140138 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:36.140051 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 27 20:40:36.140862 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:36.140842 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-nzl26\"" Jul 27 20:40:36.147745 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:36.147716 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lbx57"] Jul 27 20:40:36.193257 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:36.193212 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvg6d\" (UniqueName: \"kubernetes.io/projected/30871d9e-eee0-47b9-aa1b-6963cbaa54ea-kube-api-access-zvg6d\") pod \"openshift-lws-operator-fd8ccff4c-lbx57\" (UID: \"30871d9e-eee0-47b9-aa1b-6963cbaa54ea\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lbx57" Jul 27 20:40:36.193429 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:36.193270 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30871d9e-eee0-47b9-aa1b-6963cbaa54ea-tmp\") pod \"openshift-lws-operator-fd8ccff4c-lbx57\" (UID: \"30871d9e-eee0-47b9-aa1b-6963cbaa54ea\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lbx57" Jul 27 20:40:36.294091 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:36.294054 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvg6d\" (UniqueName: \"kubernetes.io/projected/30871d9e-eee0-47b9-aa1b-6963cbaa54ea-kube-api-access-zvg6d\") pod \"openshift-lws-operator-fd8ccff4c-lbx57\" (UID: \"30871d9e-eee0-47b9-aa1b-6963cbaa54ea\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lbx57" Jul 27 20:40:36.294256 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:36.294106 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30871d9e-eee0-47b9-aa1b-6963cbaa54ea-tmp\") pod \"openshift-lws-operator-fd8ccff4c-lbx57\" (UID: \"30871d9e-eee0-47b9-aa1b-6963cbaa54ea\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lbx57" Jul 27 20:40:36.294464 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:36.294445 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30871d9e-eee0-47b9-aa1b-6963cbaa54ea-tmp\") pod \"openshift-lws-operator-fd8ccff4c-lbx57\" (UID: \"30871d9e-eee0-47b9-aa1b-6963cbaa54ea\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lbx57" Jul 27 20:40:36.302536 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:36.302500 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvg6d\" (UniqueName: \"kubernetes.io/projected/30871d9e-eee0-47b9-aa1b-6963cbaa54ea-kube-api-access-zvg6d\") pod \"openshift-lws-operator-fd8ccff4c-lbx57\" (UID: \"30871d9e-eee0-47b9-aa1b-6963cbaa54ea\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lbx57" Jul 27 20:40:36.447188 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:36.447082 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lbx57" Jul 27 20:40:36.577917 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:36.577895 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lbx57"] Jul 27 20:40:36.580068 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:40:36.580040 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30871d9e_eee0_47b9_aa1b_6963cbaa54ea.slice/crio-6ec1085eb23209b6b97435e12a9aec70327c616df096366109662e45f1829c29 WatchSource:0}: Error finding container 6ec1085eb23209b6b97435e12a9aec70327c616df096366109662e45f1829c29: Status 404 returned error can't find the container with id 6ec1085eb23209b6b97435e12a9aec70327c616df096366109662e45f1829c29 Jul 27 20:40:37.461528 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:37.461489 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lbx57" event={"ID":"30871d9e-eee0-47b9-aa1b-6963cbaa54ea","Type":"ContainerStarted","Data":"6ec1085eb23209b6b97435e12a9aec70327c616df096366109662e45f1829c29"} Jul 27 20:40:40.477912 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:40.477870 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lbx57" event={"ID":"30871d9e-eee0-47b9-aa1b-6963cbaa54ea","Type":"ContainerStarted","Data":"651326e0120b5310362bc5025afb2f5273dda917ae6b4b314773cd823d3bdb41"} Jul 27 20:40:40.496269 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:40.496213 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lbx57" podStartSLOduration=1.622415425 podStartE2EDuration="4.4961971s" totalImagesPullingTime="2.873781675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:40:40.494426891 +0000 UTC m=+370.230407180" watchObservedRunningTime="2026-07-27 20:40:40.4961971 +0000 UTC m=+370.232177387" Jul 27 20:40:43.587550 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:43.587517 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6nwqs"] Jul 27 20:40:43.590012 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:43.589987 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6nwqs" Jul 27 20:40:43.592394 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:43.592372 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ntvl4\"" Jul 27 20:40:43.592652 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:43.592637 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 27 20:40:43.593321 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:43.593302 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 27 20:40:43.602257 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:43.602234 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6nwqs"] Jul 27 20:40:43.659249 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:43.659207 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4vrk\" (UniqueName: \"kubernetes.io/projected/0a4b7097-1111-4168-82af-f70e784fbef9-kube-api-access-c4vrk\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6nwqs\" (UID: \"0a4b7097-1111-4168-82af-f70e784fbef9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6nwqs" Jul 27 20:40:43.659461 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:43.659273 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a4b7097-1111-4168-82af-f70e784fbef9-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6nwqs\" (UID: \"0a4b7097-1111-4168-82af-f70e784fbef9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6nwqs" Jul 27 20:40:43.659461 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:43.659379 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a4b7097-1111-4168-82af-f70e784fbef9-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6nwqs\" (UID: \"0a4b7097-1111-4168-82af-f70e784fbef9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6nwqs" Jul 27 20:40:43.760848 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:43.760814 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a4b7097-1111-4168-82af-f70e784fbef9-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6nwqs\" (UID: \"0a4b7097-1111-4168-82af-f70e784fbef9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6nwqs" Jul 27 20:40:43.761044 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:43.760864 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a4b7097-1111-4168-82af-f70e784fbef9-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6nwqs\" (UID: \"0a4b7097-1111-4168-82af-f70e784fbef9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6nwqs" Jul 27 20:40:43.761044 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:43.760902 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4vrk\" (UniqueName: \"kubernetes.io/projected/0a4b7097-1111-4168-82af-f70e784fbef9-kube-api-access-c4vrk\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6nwqs\" (UID: \"0a4b7097-1111-4168-82af-f70e784fbef9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6nwqs" Jul 27 20:40:43.761261 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:43.761235 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a4b7097-1111-4168-82af-f70e784fbef9-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6nwqs\" (UID: \"0a4b7097-1111-4168-82af-f70e784fbef9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6nwqs" Jul 27 20:40:43.761349 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:43.761307 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a4b7097-1111-4168-82af-f70e784fbef9-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6nwqs\" (UID: \"0a4b7097-1111-4168-82af-f70e784fbef9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6nwqs" Jul 27 20:40:43.768745 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:43.768717 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4vrk\" (UniqueName: \"kubernetes.io/projected/0a4b7097-1111-4168-82af-f70e784fbef9-kube-api-access-c4vrk\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6nwqs\" (UID: \"0a4b7097-1111-4168-82af-f70e784fbef9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6nwqs" Jul 27 20:40:43.899492 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:43.899399 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6nwqs" Jul 27 20:40:44.035244 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:44.035214 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6nwqs"] Jul 27 20:40:44.037629 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:40:44.037583 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a4b7097_1111_4168_82af_f70e784fbef9.slice/crio-e73816955f108f1e9ef2d9aa7d53791641d6954a9091e5ccf698f710393e3b7d WatchSource:0}: Error finding container e73816955f108f1e9ef2d9aa7d53791641d6954a9091e5ccf698f710393e3b7d: Status 404 returned error can't find the container with id e73816955f108f1e9ef2d9aa7d53791641d6954a9091e5ccf698f710393e3b7d Jul 27 20:40:44.497370 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:44.497336 2570 generic.go:356] "Generic (PLEG): container finished" podID="0a4b7097-1111-4168-82af-f70e784fbef9" containerID="d52fab867a78f53e0b62ff5205fa9e8aa27af5235ddfca2ee29def1f2c61684b" exitCode=0 Jul 27 20:40:44.497562 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:44.497430 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6nwqs" event={"ID":"0a4b7097-1111-4168-82af-f70e784fbef9","Type":"ContainerDied","Data":"d52fab867a78f53e0b62ff5205fa9e8aa27af5235ddfca2ee29def1f2c61684b"} Jul 27 20:40:44.497562 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:44.497515 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6nwqs" event={"ID":"0a4b7097-1111-4168-82af-f70e784fbef9","Type":"ContainerStarted","Data":"e73816955f108f1e9ef2d9aa7d53791641d6954a9091e5ccf698f710393e3b7d"} Jul 27 20:40:45.503874 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:45.503838 2570 generic.go:356] "Generic (PLEG): container finished" podID="0a4b7097-1111-4168-82af-f70e784fbef9" containerID="3974448f3a85318ffee9734978e3d8c63be43099104887cea4218adfd3c37368" exitCode=0 Jul 27 20:40:45.504259 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:45.503893 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6nwqs" event={"ID":"0a4b7097-1111-4168-82af-f70e784fbef9","Type":"ContainerDied","Data":"3974448f3a85318ffee9734978e3d8c63be43099104887cea4218adfd3c37368"} Jul 27 20:40:46.510657 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:46.510621 2570 generic.go:356] "Generic (PLEG): container finished" podID="0a4b7097-1111-4168-82af-f70e784fbef9" containerID="6acab07b41768ce87a9e9d19f76213141c0f650987ee4e635e3197eb8f60f27f" exitCode=0 Jul 27 20:40:46.511035 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:46.510673 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6nwqs" event={"ID":"0a4b7097-1111-4168-82af-f70e784fbef9","Type":"ContainerDied","Data":"6acab07b41768ce87a9e9d19f76213141c0f650987ee4e635e3197eb8f60f27f"} Jul 27 20:40:47.654971 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:47.654943 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6nwqs" Jul 27 20:40:47.698230 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:47.698199 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a4b7097-1111-4168-82af-f70e784fbef9-util\") pod \"0a4b7097-1111-4168-82af-f70e784fbef9\" (UID: \"0a4b7097-1111-4168-82af-f70e784fbef9\") " Jul 27 20:40:47.698230 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:47.698235 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a4b7097-1111-4168-82af-f70e784fbef9-bundle\") pod \"0a4b7097-1111-4168-82af-f70e784fbef9\" (UID: \"0a4b7097-1111-4168-82af-f70e784fbef9\") " Jul 27 20:40:47.698431 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:47.698283 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4vrk\" (UniqueName: \"kubernetes.io/projected/0a4b7097-1111-4168-82af-f70e784fbef9-kube-api-access-c4vrk\") pod \"0a4b7097-1111-4168-82af-f70e784fbef9\" (UID: \"0a4b7097-1111-4168-82af-f70e784fbef9\") " Jul 27 20:40:47.699017 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:47.698988 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a4b7097-1111-4168-82af-f70e784fbef9-bundle" (OuterVolumeSpecName: "bundle") pod "0a4b7097-1111-4168-82af-f70e784fbef9" (UID: "0a4b7097-1111-4168-82af-f70e784fbef9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 20:40:47.700576 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:47.700553 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a4b7097-1111-4168-82af-f70e784fbef9-kube-api-access-c4vrk" (OuterVolumeSpecName: "kube-api-access-c4vrk") pod "0a4b7097-1111-4168-82af-f70e784fbef9" (UID: "0a4b7097-1111-4168-82af-f70e784fbef9"). InnerVolumeSpecName "kube-api-access-c4vrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:40:47.705253 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:47.705215 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a4b7097-1111-4168-82af-f70e784fbef9-util" (OuterVolumeSpecName: "util") pod "0a4b7097-1111-4168-82af-f70e784fbef9" (UID: "0a4b7097-1111-4168-82af-f70e784fbef9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 20:40:47.799221 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:47.799126 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a4b7097-1111-4168-82af-f70e784fbef9-util\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:40:47.799221 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:47.799162 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a4b7097-1111-4168-82af-f70e784fbef9-bundle\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:40:47.799221 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:47.799173 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4vrk\" (UniqueName: \"kubernetes.io/projected/0a4b7097-1111-4168-82af-f70e784fbef9-kube-api-access-c4vrk\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:40:48.522114 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:48.522069 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6nwqs" event={"ID":"0a4b7097-1111-4168-82af-f70e784fbef9","Type":"ContainerDied","Data":"e73816955f108f1e9ef2d9aa7d53791641d6954a9091e5ccf698f710393e3b7d"} Jul 27 20:40:48.522302 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:48.522138 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e73816955f108f1e9ef2d9aa7d53791641d6954a9091e5ccf698f710393e3b7d" Jul 27 20:40:48.522302 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:48.522083 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6nwqs" Jul 27 20:40:56.347918 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:56.347875 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5767d7b464-bmvmq"] Jul 27 20:40:56.348390 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:56.348332 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a4b7097-1111-4168-82af-f70e784fbef9" containerName="util" Jul 27 20:40:56.348390 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:56.348350 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a4b7097-1111-4168-82af-f70e784fbef9" containerName="util" Jul 27 20:40:56.348390 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:56.348370 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a4b7097-1111-4168-82af-f70e784fbef9" containerName="pull" Jul 27 20:40:56.348390 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:56.348378 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a4b7097-1111-4168-82af-f70e784fbef9" containerName="pull" Jul 27 20:40:56.348623 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:56.348405 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a4b7097-1111-4168-82af-f70e784fbef9" containerName="extract" Jul 27 20:40:56.348623 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:56.348414 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a4b7097-1111-4168-82af-f70e784fbef9" containerName="extract" Jul 27 20:40:56.348623 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:56.348487 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a4b7097-1111-4168-82af-f70e784fbef9" containerName="extract" Jul 27 20:40:56.350878 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:56.350857 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5767d7b464-bmvmq" Jul 27 20:40:56.353501 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:56.353481 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8slmf\"" Jul 27 20:40:56.353890 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:56.353871 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 27 20:40:56.354152 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:56.354133 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 27 20:40:56.354224 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:56.354202 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 27 20:40:56.354281 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:56.354232 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 27 20:40:56.373036 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:56.372993 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5767d7b464-bmvmq"] Jul 27 20:40:56.468885 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:56.468845 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/181cb227-a222-41b5-b8dd-f6270ba914eb-manifests\") pod \"opendatahub-operator-controller-manager-5767d7b464-bmvmq\" (UID: \"181cb227-a222-41b5-b8dd-f6270ba914eb\") " pod="opendatahub/opendatahub-operator-controller-manager-5767d7b464-bmvmq" Jul 27 20:40:56.469088 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:56.468920 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcgdf\" (UniqueName: \"kubernetes.io/projected/181cb227-a222-41b5-b8dd-f6270ba914eb-kube-api-access-vcgdf\") pod \"opendatahub-operator-controller-manager-5767d7b464-bmvmq\" (UID: \"181cb227-a222-41b5-b8dd-f6270ba914eb\") " pod="opendatahub/opendatahub-operator-controller-manager-5767d7b464-bmvmq" Jul 27 20:40:56.469088 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:56.468960 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/181cb227-a222-41b5-b8dd-f6270ba914eb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5767d7b464-bmvmq\" (UID: \"181cb227-a222-41b5-b8dd-f6270ba914eb\") " pod="opendatahub/opendatahub-operator-controller-manager-5767d7b464-bmvmq" Jul 27 20:40:56.469088 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:56.469033 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/181cb227-a222-41b5-b8dd-f6270ba914eb-tmp\") pod \"opendatahub-operator-controller-manager-5767d7b464-bmvmq\" (UID: \"181cb227-a222-41b5-b8dd-f6270ba914eb\") " pod="opendatahub/opendatahub-operator-controller-manager-5767d7b464-bmvmq" Jul 27 20:40:56.469310 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:56.469093 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/181cb227-a222-41b5-b8dd-f6270ba914eb-webhook-cert\") pod \"opendatahub-operator-controller-manager-5767d7b464-bmvmq\" (UID: \"181cb227-a222-41b5-b8dd-f6270ba914eb\") " pod="opendatahub/opendatahub-operator-controller-manager-5767d7b464-bmvmq" Jul 27 20:40:56.570303 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:56.570262 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/181cb227-a222-41b5-b8dd-f6270ba914eb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5767d7b464-bmvmq\" (UID: \"181cb227-a222-41b5-b8dd-f6270ba914eb\") " pod="opendatahub/opendatahub-operator-controller-manager-5767d7b464-bmvmq" Jul 27 20:40:56.570491 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:56.570314 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/181cb227-a222-41b5-b8dd-f6270ba914eb-tmp\") pod \"opendatahub-operator-controller-manager-5767d7b464-bmvmq\" (UID: \"181cb227-a222-41b5-b8dd-f6270ba914eb\") " pod="opendatahub/opendatahub-operator-controller-manager-5767d7b464-bmvmq" Jul 27 20:40:56.570491 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:56.570357 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/181cb227-a222-41b5-b8dd-f6270ba914eb-webhook-cert\") pod \"opendatahub-operator-controller-manager-5767d7b464-bmvmq\" (UID: \"181cb227-a222-41b5-b8dd-f6270ba914eb\") " pod="opendatahub/opendatahub-operator-controller-manager-5767d7b464-bmvmq" Jul 27 20:40:56.570491 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:56.570393 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/181cb227-a222-41b5-b8dd-f6270ba914eb-manifests\") pod \"opendatahub-operator-controller-manager-5767d7b464-bmvmq\" (UID: \"181cb227-a222-41b5-b8dd-f6270ba914eb\") " pod="opendatahub/opendatahub-operator-controller-manager-5767d7b464-bmvmq" Jul 27 20:40:56.570491 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:56.570422 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcgdf\" (UniqueName: \"kubernetes.io/projected/181cb227-a222-41b5-b8dd-f6270ba914eb-kube-api-access-vcgdf\") pod \"opendatahub-operator-controller-manager-5767d7b464-bmvmq\" (UID: \"181cb227-a222-41b5-b8dd-f6270ba914eb\") " pod="opendatahub/opendatahub-operator-controller-manager-5767d7b464-bmvmq" Jul 27 20:40:56.570900 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:56.570868 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/181cb227-a222-41b5-b8dd-f6270ba914eb-manifests\") pod \"opendatahub-operator-controller-manager-5767d7b464-bmvmq\" (UID: \"181cb227-a222-41b5-b8dd-f6270ba914eb\") " pod="opendatahub/opendatahub-operator-controller-manager-5767d7b464-bmvmq" Jul 27 20:40:56.572995 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:56.572963 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/181cb227-a222-41b5-b8dd-f6270ba914eb-tmp\") pod \"opendatahub-operator-controller-manager-5767d7b464-bmvmq\" (UID: \"181cb227-a222-41b5-b8dd-f6270ba914eb\") " pod="opendatahub/opendatahub-operator-controller-manager-5767d7b464-bmvmq" Jul 27 20:40:56.573114 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:56.573085 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/181cb227-a222-41b5-b8dd-f6270ba914eb-webhook-cert\") pod \"opendatahub-operator-controller-manager-5767d7b464-bmvmq\" (UID: \"181cb227-a222-41b5-b8dd-f6270ba914eb\") " pod="opendatahub/opendatahub-operator-controller-manager-5767d7b464-bmvmq" Jul 27 20:40:56.573174 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:56.573084 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/181cb227-a222-41b5-b8dd-f6270ba914eb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5767d7b464-bmvmq\" (UID: \"181cb227-a222-41b5-b8dd-f6270ba914eb\") " pod="opendatahub/opendatahub-operator-controller-manager-5767d7b464-bmvmq" Jul 27 20:40:56.583837 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:56.583801 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcgdf\" (UniqueName: \"kubernetes.io/projected/181cb227-a222-41b5-b8dd-f6270ba914eb-kube-api-access-vcgdf\") pod \"opendatahub-operator-controller-manager-5767d7b464-bmvmq\" (UID: \"181cb227-a222-41b5-b8dd-f6270ba914eb\") " pod="opendatahub/opendatahub-operator-controller-manager-5767d7b464-bmvmq" Jul 27 20:40:56.662756 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:56.662659 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5767d7b464-bmvmq" Jul 27 20:40:56.801808 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:56.801783 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5767d7b464-bmvmq"] Jul 27 20:40:56.804342 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:40:56.804315 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod181cb227_a222_41b5_b8dd_f6270ba914eb.slice/crio-71a5ba4ab8595821d818f55852066d21be81fc0671f2d09db1ac8b3bc99b53a3 WatchSource:0}: Error finding container 71a5ba4ab8595821d818f55852066d21be81fc0671f2d09db1ac8b3bc99b53a3: Status 404 returned error can't find the container with id 71a5ba4ab8595821d818f55852066d21be81fc0671f2d09db1ac8b3bc99b53a3 Jul 27 20:40:57.564041 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:40:57.564000 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5767d7b464-bmvmq" event={"ID":"181cb227-a222-41b5-b8dd-f6270ba914eb","Type":"ContainerStarted","Data":"71a5ba4ab8595821d818f55852066d21be81fc0671f2d09db1ac8b3bc99b53a3"} Jul 27 20:41:00.581270 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:00.581217 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5767d7b464-bmvmq" event={"ID":"181cb227-a222-41b5-b8dd-f6270ba914eb","Type":"ContainerStarted","Data":"cf75e130962c6a6bb5479f24e0e42ee5557464c2d4e5369054ce98ca9ef15816"} Jul 27 20:41:01.587197 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:01.587160 2570 generic.go:356] "Generic (PLEG): container finished" podID="181cb227-a222-41b5-b8dd-f6270ba914eb" containerID="cf75e130962c6a6bb5479f24e0e42ee5557464c2d4e5369054ce98ca9ef15816" exitCode=0 Jul 27 20:41:01.587591 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:01.587238 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5767d7b464-bmvmq" event={"ID":"181cb227-a222-41b5-b8dd-f6270ba914eb","Type":"ContainerDied","Data":"cf75e130962c6a6bb5479f24e0e42ee5557464c2d4e5369054ce98ca9ef15816"} Jul 27 20:41:02.594549 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:02.594512 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5767d7b464-bmvmq" event={"ID":"181cb227-a222-41b5-b8dd-f6270ba914eb","Type":"ContainerStarted","Data":"0f3b0902c600a127a4bcaa1f42b8234f6db2c49601850024ac6ccf69f6410f66"} Jul 27 20:41:02.594995 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:02.594755 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5767d7b464-bmvmq" Jul 27 20:41:02.680022 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:02.679936 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5767d7b464-bmvmq" podStartSLOduration=2.407848304 podStartE2EDuration="6.679918916s" totalImagesPullingTime="3.212112821s" totalInitContainerRuntime="1.059957791s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:40:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:41:02.651502278 +0000 UTC m=+392.387482578" watchObservedRunningTime="2026-07-27 20:41:02.679918916 +0000 UTC m=+392.415899202" Jul 27 20:41:13.602317 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:13.602284 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5767d7b464-bmvmq" Jul 27 20:41:16.138197 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:16.138160 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-57b999c94c-rx7lh"] Jul 27 20:41:16.150395 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:16.150364 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-57b999c94c-rx7lh" Jul 27 20:41:16.153382 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:16.153361 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-r6j5r\"" Jul 27 20:41:16.154331 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:16.154307 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 27 20:41:16.154435 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:16.154374 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 27 20:41:16.154435 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:16.154418 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 27 20:41:16.154535 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:16.154498 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 27 20:41:16.165863 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:16.165830 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-57b999c94c-rx7lh"] Jul 27 20:41:16.246721 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:16.246684 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whfjl\" (UniqueName: \"kubernetes.io/projected/f242764c-560b-4ab8-a4c3-23d74e64a6d2-kube-api-access-whfjl\") pod \"kube-auth-proxy-57b999c94c-rx7lh\" (UID: \"f242764c-560b-4ab8-a4c3-23d74e64a6d2\") " pod="openshift-ingress/kube-auth-proxy-57b999c94c-rx7lh" Jul 27 20:41:16.246910 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:16.246756 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f242764c-560b-4ab8-a4c3-23d74e64a6d2-tls-certs\") pod \"kube-auth-proxy-57b999c94c-rx7lh\" (UID: \"f242764c-560b-4ab8-a4c3-23d74e64a6d2\") " pod="openshift-ingress/kube-auth-proxy-57b999c94c-rx7lh" Jul 27 20:41:16.246910 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:16.246787 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f242764c-560b-4ab8-a4c3-23d74e64a6d2-tmp\") pod \"kube-auth-proxy-57b999c94c-rx7lh\" (UID: \"f242764c-560b-4ab8-a4c3-23d74e64a6d2\") " pod="openshift-ingress/kube-auth-proxy-57b999c94c-rx7lh" Jul 27 20:41:16.308167 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:16.308129 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-5vxxm"] Jul 27 20:41:16.310994 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:16.310971 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-5vxxm" Jul 27 20:41:16.313682 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:16.313454 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 27 20:41:16.313682 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:16.313660 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-chbtd\"" Jul 27 20:41:16.327340 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:16.327310 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-5vxxm"] Jul 27 20:41:16.348063 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:16.348028 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f242764c-560b-4ab8-a4c3-23d74e64a6d2-tls-certs\") pod \"kube-auth-proxy-57b999c94c-rx7lh\" (UID: \"f242764c-560b-4ab8-a4c3-23d74e64a6d2\") " pod="openshift-ingress/kube-auth-proxy-57b999c94c-rx7lh" Jul 27 20:41:16.348063 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:16.348072 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f242764c-560b-4ab8-a4c3-23d74e64a6d2-tmp\") pod \"kube-auth-proxy-57b999c94c-rx7lh\" (UID: \"f242764c-560b-4ab8-a4c3-23d74e64a6d2\") " pod="openshift-ingress/kube-auth-proxy-57b999c94c-rx7lh" Jul 27 20:41:16.348297 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:16.348172 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whfjl\" (UniqueName: \"kubernetes.io/projected/f242764c-560b-4ab8-a4c3-23d74e64a6d2-kube-api-access-whfjl\") pod \"kube-auth-proxy-57b999c94c-rx7lh\" (UID: \"f242764c-560b-4ab8-a4c3-23d74e64a6d2\") " pod="openshift-ingress/kube-auth-proxy-57b999c94c-rx7lh" Jul 27 20:41:16.350698 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:16.350674 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f242764c-560b-4ab8-a4c3-23d74e64a6d2-tmp\") pod \"kube-auth-proxy-57b999c94c-rx7lh\" (UID: \"f242764c-560b-4ab8-a4c3-23d74e64a6d2\") " pod="openshift-ingress/kube-auth-proxy-57b999c94c-rx7lh" Jul 27 20:41:16.350824 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:16.350804 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f242764c-560b-4ab8-a4c3-23d74e64a6d2-tls-certs\") pod \"kube-auth-proxy-57b999c94c-rx7lh\" (UID: \"f242764c-560b-4ab8-a4c3-23d74e64a6d2\") " pod="openshift-ingress/kube-auth-proxy-57b999c94c-rx7lh" Jul 27 20:41:16.367349 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:16.367325 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whfjl\" (UniqueName: \"kubernetes.io/projected/f242764c-560b-4ab8-a4c3-23d74e64a6d2-kube-api-access-whfjl\") pod \"kube-auth-proxy-57b999c94c-rx7lh\" (UID: \"f242764c-560b-4ab8-a4c3-23d74e64a6d2\") " pod="openshift-ingress/kube-auth-proxy-57b999c94c-rx7lh" Jul 27 20:41:16.449100 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:16.448996 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a82dfdde-1dc6-4286-9e7d-d8cc6c18c2fc-cert\") pod \"odh-model-controller-7998c67dc4-5vxxm\" (UID: \"a82dfdde-1dc6-4286-9e7d-d8cc6c18c2fc\") " pod="opendatahub/odh-model-controller-7998c67dc4-5vxxm" Jul 27 20:41:16.449100 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:16.449033 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6s92\" (UniqueName: \"kubernetes.io/projected/a82dfdde-1dc6-4286-9e7d-d8cc6c18c2fc-kube-api-access-r6s92\") pod \"odh-model-controller-7998c67dc4-5vxxm\" (UID: \"a82dfdde-1dc6-4286-9e7d-d8cc6c18c2fc\") " pod="opendatahub/odh-model-controller-7998c67dc4-5vxxm" Jul 27 20:41:16.460991 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:16.460960 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-57b999c94c-rx7lh" Jul 27 20:41:16.549928 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:16.549881 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a82dfdde-1dc6-4286-9e7d-d8cc6c18c2fc-cert\") pod \"odh-model-controller-7998c67dc4-5vxxm\" (UID: \"a82dfdde-1dc6-4286-9e7d-d8cc6c18c2fc\") " pod="opendatahub/odh-model-controller-7998c67dc4-5vxxm" Jul 27 20:41:16.549928 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:16.549922 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6s92\" (UniqueName: \"kubernetes.io/projected/a82dfdde-1dc6-4286-9e7d-d8cc6c18c2fc-kube-api-access-r6s92\") pod \"odh-model-controller-7998c67dc4-5vxxm\" (UID: \"a82dfdde-1dc6-4286-9e7d-d8cc6c18c2fc\") " pod="opendatahub/odh-model-controller-7998c67dc4-5vxxm" Jul 27 20:41:16.552389 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:16.552363 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a82dfdde-1dc6-4286-9e7d-d8cc6c18c2fc-cert\") pod \"odh-model-controller-7998c67dc4-5vxxm\" (UID: \"a82dfdde-1dc6-4286-9e7d-d8cc6c18c2fc\") " pod="opendatahub/odh-model-controller-7998c67dc4-5vxxm" Jul 27 20:41:16.559097 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:16.559066 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6s92\" (UniqueName: \"kubernetes.io/projected/a82dfdde-1dc6-4286-9e7d-d8cc6c18c2fc-kube-api-access-r6s92\") pod \"odh-model-controller-7998c67dc4-5vxxm\" (UID: \"a82dfdde-1dc6-4286-9e7d-d8cc6c18c2fc\") " pod="opendatahub/odh-model-controller-7998c67dc4-5vxxm" Jul 27 20:41:16.617067 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:16.617040 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-57b999c94c-rx7lh"] Jul 27 20:41:16.620299 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:41:16.620248 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf242764c_560b_4ab8_a4c3_23d74e64a6d2.slice/crio-e1fe9062546130ffd75a6f2105322768770933bff414c780e126e9ef749c323a WatchSource:0}: Error finding container e1fe9062546130ffd75a6f2105322768770933bff414c780e126e9ef749c323a: Status 404 returned error can't find the container with id e1fe9062546130ffd75a6f2105322768770933bff414c780e126e9ef749c323a Jul 27 20:41:16.624745 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:16.624721 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-5vxxm" Jul 27 20:41:16.660028 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:16.659979 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-57b999c94c-rx7lh" event={"ID":"f242764c-560b-4ab8-a4c3-23d74e64a6d2","Type":"ContainerStarted","Data":"e1fe9062546130ffd75a6f2105322768770933bff414c780e126e9ef749c323a"} Jul 27 20:41:16.780510 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:16.780472 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-5vxxm"] Jul 27 20:41:16.783855 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:41:16.783816 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda82dfdde_1dc6_4286_9e7d_d8cc6c18c2fc.slice/crio-1e7a33c987d0392d491736cc3df0f2ca0b9e31e1e9d603f64f70c73ed3852658 WatchSource:0}: Error finding container 1e7a33c987d0392d491736cc3df0f2ca0b9e31e1e9d603f64f70c73ed3852658: Status 404 returned error can't find the container with id 1e7a33c987d0392d491736cc3df0f2ca0b9e31e1e9d603f64f70c73ed3852658 Jul 27 20:41:17.376236 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.376198 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mssp2"] Jul 27 20:41:17.393879 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.393835 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk"] Jul 27 20:41:17.394076 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.393965 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-mssp2" Jul 27 20:41:17.398147 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.397281 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mssp2"] Jul 27 20:41:17.398147 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.397401 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk" Jul 27 20:41:17.398588 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.398564 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 27 20:41:17.398915 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.398896 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-r5v8n\"" Jul 27 20:41:17.401850 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.400364 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 27 20:41:17.401850 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.400653 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 27 20:41:17.401850 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.400822 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 27 20:41:17.401850 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.400927 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 27 20:41:17.401850 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.401043 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-nhxhv\"" Jul 27 20:41:17.413018 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.412429 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk"] Jul 27 20:41:17.559377 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.559337 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0b672fab-a969-445e-89e4-b1668da529b8-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-6psfk\" (UID: \"0b672fab-a969-445e-89e4-b1668da529b8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk" Jul 27 20:41:17.559578 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.559402 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfc930c7-c597-440b-a01e-b07a05ed50ff-tls-certs\") pod \"model-serving-api-54d89bd5fc-mssp2\" (UID: \"dfc930c7-c597-440b-a01e-b07a05ed50ff\") " pod="opendatahub/model-serving-api-54d89bd5fc-mssp2" Jul 27 20:41:17.559578 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.559434 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0b672fab-a969-445e-89e4-b1668da529b8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-6psfk\" (UID: \"0b672fab-a969-445e-89e4-b1668da529b8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk" Jul 27 20:41:17.559578 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.559489 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt2gt\" (UniqueName: \"kubernetes.io/projected/0b672fab-a969-445e-89e4-b1668da529b8-kube-api-access-nt2gt\") pod \"istiod-openshift-gateway-7b7995d5cd-6psfk\" (UID: \"0b672fab-a969-445e-89e4-b1668da529b8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk" Jul 27 20:41:17.559578 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.559526 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5kb4\" (UniqueName: \"kubernetes.io/projected/dfc930c7-c597-440b-a01e-b07a05ed50ff-kube-api-access-c5kb4\") pod \"model-serving-api-54d89bd5fc-mssp2\" (UID: \"dfc930c7-c597-440b-a01e-b07a05ed50ff\") " pod="opendatahub/model-serving-api-54d89bd5fc-mssp2" Jul 27 20:41:17.559578 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.559554 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b672fab-a969-445e-89e4-b1668da529b8-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-6psfk\" (UID: \"0b672fab-a969-445e-89e4-b1668da529b8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk" Jul 27 20:41:17.559578 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.559575 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b672fab-a969-445e-89e4-b1668da529b8-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-6psfk\" (UID: \"0b672fab-a969-445e-89e4-b1668da529b8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk" Jul 27 20:41:17.559895 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.559635 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0b672fab-a969-445e-89e4-b1668da529b8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-6psfk\" (UID: \"0b672fab-a969-445e-89e4-b1668da529b8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk" Jul 27 20:41:17.559895 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.559726 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0b672fab-a969-445e-89e4-b1668da529b8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-6psfk\" (UID: \"0b672fab-a969-445e-89e4-b1668da529b8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk" Jul 27 20:41:17.661996 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.660852 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt2gt\" (UniqueName: \"kubernetes.io/projected/0b672fab-a969-445e-89e4-b1668da529b8-kube-api-access-nt2gt\") pod \"istiod-openshift-gateway-7b7995d5cd-6psfk\" (UID: \"0b672fab-a969-445e-89e4-b1668da529b8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk" Jul 27 20:41:17.661996 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.660918 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5kb4\" (UniqueName: \"kubernetes.io/projected/dfc930c7-c597-440b-a01e-b07a05ed50ff-kube-api-access-c5kb4\") pod \"model-serving-api-54d89bd5fc-mssp2\" (UID: \"dfc930c7-c597-440b-a01e-b07a05ed50ff\") " pod="opendatahub/model-serving-api-54d89bd5fc-mssp2" Jul 27 20:41:17.661996 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.660974 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b672fab-a969-445e-89e4-b1668da529b8-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-6psfk\" (UID: \"0b672fab-a969-445e-89e4-b1668da529b8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk" Jul 27 20:41:17.661996 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.661008 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b672fab-a969-445e-89e4-b1668da529b8-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-6psfk\" (UID: \"0b672fab-a969-445e-89e4-b1668da529b8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk" Jul 27 20:41:17.661996 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.661036 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0b672fab-a969-445e-89e4-b1668da529b8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-6psfk\" (UID: \"0b672fab-a969-445e-89e4-b1668da529b8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk" Jul 27 20:41:17.661996 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.661146 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0b672fab-a969-445e-89e4-b1668da529b8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-6psfk\" (UID: \"0b672fab-a969-445e-89e4-b1668da529b8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk" Jul 27 20:41:17.661996 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.661244 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0b672fab-a969-445e-89e4-b1668da529b8-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-6psfk\" (UID: \"0b672fab-a969-445e-89e4-b1668da529b8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk" Jul 27 20:41:17.661996 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.661310 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfc930c7-c597-440b-a01e-b07a05ed50ff-tls-certs\") pod \"model-serving-api-54d89bd5fc-mssp2\" (UID: \"dfc930c7-c597-440b-a01e-b07a05ed50ff\") " pod="opendatahub/model-serving-api-54d89bd5fc-mssp2" Jul 27 20:41:17.661996 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.661333 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0b672fab-a969-445e-89e4-b1668da529b8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-6psfk\" (UID: \"0b672fab-a969-445e-89e4-b1668da529b8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk" Jul 27 20:41:17.664941 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.664146 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0b672fab-a969-445e-89e4-b1668da529b8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-6psfk\" (UID: \"0b672fab-a969-445e-89e4-b1668da529b8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk" Jul 27 20:41:17.664941 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:41:17.664896 2570 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 27 20:41:17.665072 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:41:17.664974 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfc930c7-c597-440b-a01e-b07a05ed50ff-tls-certs podName:dfc930c7-c597-440b-a01e-b07a05ed50ff nodeName:}" failed. No retries permitted until 2026-07-27 20:41:18.164953214 +0000 UTC m=+407.900933492 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/dfc930c7-c597-440b-a01e-b07a05ed50ff-tls-certs") pod "model-serving-api-54d89bd5fc-mssp2" (UID: "dfc930c7-c597-440b-a01e-b07a05ed50ff") : secret "model-serving-api-tls" not found Jul 27 20:41:17.666752 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.666702 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0b672fab-a969-445e-89e4-b1668da529b8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-6psfk\" (UID: \"0b672fab-a969-445e-89e4-b1668da529b8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk" Jul 27 20:41:17.668051 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.668018 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0b672fab-a969-445e-89e4-b1668da529b8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-6psfk\" (UID: \"0b672fab-a969-445e-89e4-b1668da529b8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk" Jul 27 20:41:17.671319 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.671230 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-5vxxm" event={"ID":"a82dfdde-1dc6-4286-9e7d-d8cc6c18c2fc","Type":"ContainerStarted","Data":"1e7a33c987d0392d491736cc3df0f2ca0b9e31e1e9d603f64f70c73ed3852658"} Jul 27 20:41:17.672530 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.672472 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b672fab-a969-445e-89e4-b1668da529b8-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-6psfk\" (UID: \"0b672fab-a969-445e-89e4-b1668da529b8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk" Jul 27 20:41:17.674523 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.674335 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt2gt\" (UniqueName: \"kubernetes.io/projected/0b672fab-a969-445e-89e4-b1668da529b8-kube-api-access-nt2gt\") pod \"istiod-openshift-gateway-7b7995d5cd-6psfk\" (UID: \"0b672fab-a969-445e-89e4-b1668da529b8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk" Jul 27 20:41:17.674999 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.674949 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0b672fab-a969-445e-89e4-b1668da529b8-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-6psfk\" (UID: \"0b672fab-a969-445e-89e4-b1668da529b8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk" Jul 27 20:41:17.683247 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.683215 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5kb4\" (UniqueName: \"kubernetes.io/projected/dfc930c7-c597-440b-a01e-b07a05ed50ff-kube-api-access-c5kb4\") pod \"model-serving-api-54d89bd5fc-mssp2\" (UID: \"dfc930c7-c597-440b-a01e-b07a05ed50ff\") " pod="opendatahub/model-serving-api-54d89bd5fc-mssp2" Jul 27 20:41:17.688392 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.688321 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b672fab-a969-445e-89e4-b1668da529b8-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-6psfk\" (UID: \"0b672fab-a969-445e-89e4-b1668da529b8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk" Jul 27 20:41:17.732212 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.732172 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk" Jul 27 20:41:17.948705 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:17.948652 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk"] Jul 27 20:41:17.951319 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:41:17.951272 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b672fab_a969_445e_89e4_b1668da529b8.slice/crio-531474b63333968bf8f484c177b307ed13222850ff8a23b207bdd0ababc0dff7 WatchSource:0}: Error finding container 531474b63333968bf8f484c177b307ed13222850ff8a23b207bdd0ababc0dff7: Status 404 returned error can't find the container with id 531474b63333968bf8f484c177b307ed13222850ff8a23b207bdd0ababc0dff7 Jul 27 20:41:18.167519 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:18.167389 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfc930c7-c597-440b-a01e-b07a05ed50ff-tls-certs\") pod \"model-serving-api-54d89bd5fc-mssp2\" (UID: \"dfc930c7-c597-440b-a01e-b07a05ed50ff\") " pod="opendatahub/model-serving-api-54d89bd5fc-mssp2" Jul 27 20:41:18.171633 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:18.171567 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfc930c7-c597-440b-a01e-b07a05ed50ff-tls-certs\") pod \"model-serving-api-54d89bd5fc-mssp2\" (UID: \"dfc930c7-c597-440b-a01e-b07a05ed50ff\") " pod="opendatahub/model-serving-api-54d89bd5fc-mssp2" Jul 27 20:41:18.319448 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:18.319127 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-mssp2" Jul 27 20:41:18.523637 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:18.523575 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mssp2"] Jul 27 20:41:18.528655 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:41:18.528593 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc930c7_c597_440b_a01e_b07a05ed50ff.slice/crio-553fd22d18ea682ad29854ddb865145df029cdb8e85d4701c3cd3bce08e70af3 WatchSource:0}: Error finding container 553fd22d18ea682ad29854ddb865145df029cdb8e85d4701c3cd3bce08e70af3: Status 404 returned error can't find the container with id 553fd22d18ea682ad29854ddb865145df029cdb8e85d4701c3cd3bce08e70af3 Jul 27 20:41:18.679077 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:18.678980 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-mssp2" event={"ID":"dfc930c7-c597-440b-a01e-b07a05ed50ff","Type":"ContainerStarted","Data":"553fd22d18ea682ad29854ddb865145df029cdb8e85d4701c3cd3bce08e70af3"} Jul 27 20:41:18.680721 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:18.680691 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk" event={"ID":"0b672fab-a969-445e-89e4-b1668da529b8","Type":"ContainerStarted","Data":"531474b63333968bf8f484c177b307ed13222850ff8a23b207bdd0ababc0dff7"} Jul 27 20:41:22.325142 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:22.325099 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 20:41:22.325521 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:22.325226 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 20:41:22.708808 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:22.708704 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-mssp2" event={"ID":"dfc930c7-c597-440b-a01e-b07a05ed50ff","Type":"ContainerStarted","Data":"0246883fa3ee5b2e682c5caadd4aeb7ae34bbbf7e6de998862fae8b9f9de0b8c"} Jul 27 20:41:22.708982 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:22.708807 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-mssp2" Jul 27 20:41:22.710533 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:22.710452 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-5vxxm" event={"ID":"a82dfdde-1dc6-4286-9e7d-d8cc6c18c2fc","Type":"ContainerStarted","Data":"2e738488417ac56825df8351115e65ae62bb08f415c7e68cc56f21de971ea88d"} Jul 27 20:41:22.710680 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:22.710568 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-5vxxm" Jul 27 20:41:22.712523 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:22.712496 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk" event={"ID":"0b672fab-a969-445e-89e4-b1668da529b8","Type":"ContainerStarted","Data":"229417f3ef48c5d137ba787c5f499fb70cb9e8c02449315d3ee23c9ca5082560"} Jul 27 20:41:22.712990 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:22.712953 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk" Jul 27 20:41:22.714378 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:22.714356 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-57b999c94c-rx7lh" event={"ID":"f242764c-560b-4ab8-a4c3-23d74e64a6d2","Type":"ContainerStarted","Data":"fc3c22537fa601c3d0085e0fa616e984938d61f17824879122087e0ee375d349"} Jul 27 20:41:22.728194 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:22.728131 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-mssp2" podStartSLOduration=1.975971414 podStartE2EDuration="5.728114441s" totalImagesPullingTime="3.752143027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:41:22.727700006 +0000 UTC m=+412.463680293" watchObservedRunningTime="2026-07-27 20:41:22.728114441 +0000 UTC m=+412.464094729" Jul 27 20:41:22.744348 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:22.744284 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-5vxxm" podStartSLOduration=1.2542788759999999 podStartE2EDuration="6.744265958s" totalImagesPullingTime="5.489987082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:41:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:41:22.742029249 +0000 UTC m=+412.478009553" watchObservedRunningTime="2026-07-27 20:41:22.744265958 +0000 UTC m=+412.480246249" Jul 27 20:41:22.757357 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:22.757291 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-57b999c94c-rx7lh" podStartSLOduration=2.005441876 podStartE2EDuration="6.757269888s" totalImagesPullingTime="4.751828012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:41:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:41:22.756633245 +0000 UTC m=+412.492613534" watchObservedRunningTime="2026-07-27 20:41:22.757269888 +0000 UTC m=+412.493250175" Jul 27 20:41:22.781221 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:22.781154 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk" podStartSLOduration=1.410752796 podStartE2EDuration="5.781127779s" totalImagesPullingTime="4.370374983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:41:22.779660275 +0000 UTC m=+412.515640587" watchObservedRunningTime="2026-07-27 20:41:22.781127779 +0000 UTC m=+412.517108067" Jul 27 20:41:23.721615 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:23.721564 2570 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-6psfk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 20:41:23.722000 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:23.721660 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk" podUID="0b672fab-a969-445e-89e4-b1668da529b8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 20:41:24.726592 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:24.726563 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk" Jul 27 20:41:28.030752 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:28.030708 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-xv5xd"] Jul 27 20:41:28.034616 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:28.034424 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-46drx"] Jul 27 20:41:28.034769 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:28.034714 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xv5xd" Jul 27 20:41:28.037101 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:28.037080 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-46drx" Jul 27 20:41:28.040020 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:28.039673 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 27 20:41:28.040020 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:28.039748 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-zmbdh\"" Jul 27 20:41:28.040237 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:28.040215 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-brtvt\"" Jul 27 20:41:28.040631 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:28.040588 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 27 20:41:28.058624 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:28.056947 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-46drx"] Jul 27 20:41:28.058624 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:28.057022 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-xv5xd"] Jul 27 20:41:28.075631 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:28.066398 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92cbe4b8-7401-447f-a231-e7da3f656e94-cert\") pod \"kserve-controller-manager-d6f4b65b8-46drx\" (UID: \"92cbe4b8-7401-447f-a231-e7da3f656e94\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-46drx" Jul 27 20:41:28.075631 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:28.066454 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwkdw\" (UniqueName: \"kubernetes.io/projected/92cbe4b8-7401-447f-a231-e7da3f656e94-kube-api-access-dwkdw\") pod \"kserve-controller-manager-d6f4b65b8-46drx\" (UID: \"92cbe4b8-7401-447f-a231-e7da3f656e94\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-46drx" Jul 27 20:41:28.075631 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:28.066493 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j57n2\" (UniqueName: \"kubernetes.io/projected/0a117ca9-bd46-41c8-9b0c-5791ca8cb369-kube-api-access-j57n2\") pod \"llmisvc-controller-manager-777c6bd7d5-xv5xd\" (UID: \"0a117ca9-bd46-41c8-9b0c-5791ca8cb369\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xv5xd" Jul 27 20:41:28.075631 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:28.066560 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a117ca9-bd46-41c8-9b0c-5791ca8cb369-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-xv5xd\" (UID: \"0a117ca9-bd46-41c8-9b0c-5791ca8cb369\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xv5xd" Jul 27 20:41:28.167932 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:28.167889 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a117ca9-bd46-41c8-9b0c-5791ca8cb369-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-xv5xd\" (UID: \"0a117ca9-bd46-41c8-9b0c-5791ca8cb369\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xv5xd" Jul 27 20:41:28.168115 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:28.167967 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92cbe4b8-7401-447f-a231-e7da3f656e94-cert\") pod \"kserve-controller-manager-d6f4b65b8-46drx\" (UID: \"92cbe4b8-7401-447f-a231-e7da3f656e94\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-46drx" Jul 27 20:41:28.168115 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:28.167996 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwkdw\" (UniqueName: \"kubernetes.io/projected/92cbe4b8-7401-447f-a231-e7da3f656e94-kube-api-access-dwkdw\") pod \"kserve-controller-manager-d6f4b65b8-46drx\" (UID: \"92cbe4b8-7401-447f-a231-e7da3f656e94\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-46drx" Jul 27 20:41:28.168115 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:28.168016 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j57n2\" (UniqueName: \"kubernetes.io/projected/0a117ca9-bd46-41c8-9b0c-5791ca8cb369-kube-api-access-j57n2\") pod \"llmisvc-controller-manager-777c6bd7d5-xv5xd\" (UID: \"0a117ca9-bd46-41c8-9b0c-5791ca8cb369\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xv5xd" Jul 27 20:41:28.168115 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:41:28.168044 2570 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 27 20:41:28.168271 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:41:28.168123 2570 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 27 20:41:28.168271 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:41:28.168141 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a117ca9-bd46-41c8-9b0c-5791ca8cb369-cert podName:0a117ca9-bd46-41c8-9b0c-5791ca8cb369 nodeName:}" failed. No retries permitted until 2026-07-27 20:41:28.668119052 +0000 UTC m=+418.404099336 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0a117ca9-bd46-41c8-9b0c-5791ca8cb369-cert") pod "llmisvc-controller-manager-777c6bd7d5-xv5xd" (UID: "0a117ca9-bd46-41c8-9b0c-5791ca8cb369") : secret "llmisvc-webhook-server-cert" not found Jul 27 20:41:28.168271 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:41:28.168187 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92cbe4b8-7401-447f-a231-e7da3f656e94-cert podName:92cbe4b8-7401-447f-a231-e7da3f656e94 nodeName:}" failed. No retries permitted until 2026-07-27 20:41:28.668170483 +0000 UTC m=+418.404150751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/92cbe4b8-7401-447f-a231-e7da3f656e94-cert") pod "kserve-controller-manager-d6f4b65b8-46drx" (UID: "92cbe4b8-7401-447f-a231-e7da3f656e94") : secret "kserve-webhook-server-cert" not found Jul 27 20:41:28.178642 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:28.178614 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j57n2\" (UniqueName: \"kubernetes.io/projected/0a117ca9-bd46-41c8-9b0c-5791ca8cb369-kube-api-access-j57n2\") pod \"llmisvc-controller-manager-777c6bd7d5-xv5xd\" (UID: \"0a117ca9-bd46-41c8-9b0c-5791ca8cb369\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xv5xd" Jul 27 20:41:28.178642 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:28.178644 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwkdw\" (UniqueName: \"kubernetes.io/projected/92cbe4b8-7401-447f-a231-e7da3f656e94-kube-api-access-dwkdw\") pod \"kserve-controller-manager-d6f4b65b8-46drx\" (UID: \"92cbe4b8-7401-447f-a231-e7da3f656e94\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-46drx" Jul 27 20:41:28.672825 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:28.672784 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a117ca9-bd46-41c8-9b0c-5791ca8cb369-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-xv5xd\" (UID: \"0a117ca9-bd46-41c8-9b0c-5791ca8cb369\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xv5xd" Jul 27 20:41:28.673068 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:28.672854 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92cbe4b8-7401-447f-a231-e7da3f656e94-cert\") pod \"kserve-controller-manager-d6f4b65b8-46drx\" (UID: \"92cbe4b8-7401-447f-a231-e7da3f656e94\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-46drx" Jul 27 20:41:28.675458 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:28.675426 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a117ca9-bd46-41c8-9b0c-5791ca8cb369-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-xv5xd\" (UID: \"0a117ca9-bd46-41c8-9b0c-5791ca8cb369\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xv5xd" Jul 27 20:41:28.675577 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:28.675508 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92cbe4b8-7401-447f-a231-e7da3f656e94-cert\") pod \"kserve-controller-manager-d6f4b65b8-46drx\" (UID: \"92cbe4b8-7401-447f-a231-e7da3f656e94\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-46drx" Jul 27 20:41:28.678412 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:28.678392 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-46drx" Jul 27 20:41:28.822699 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:28.822672 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-46drx"] Jul 27 20:41:28.824787 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:41:28.824754 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92cbe4b8_7401_447f_a231_e7da3f656e94.slice/crio-36b1465c00bedd8a4cb610503b36f76c359885ad555c3eb8ad1f99704d42bd82 WatchSource:0}: Error finding container 36b1465c00bedd8a4cb610503b36f76c359885ad555c3eb8ad1f99704d42bd82: Status 404 returned error can't find the container with id 36b1465c00bedd8a4cb610503b36f76c359885ad555c3eb8ad1f99704d42bd82 Jul 27 20:41:28.962199 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:28.962166 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xv5xd" Jul 27 20:41:29.118864 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:29.118766 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-xv5xd"] Jul 27 20:41:29.120975 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:41:29.120945 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0a117ca9_bd46_41c8_9b0c_5791ca8cb369.slice/crio-e9da76a3aa08694ff048ba92ae6614e529e1c590c411edbb1fdb974b25ae3dd1 WatchSource:0}: Error finding container e9da76a3aa08694ff048ba92ae6614e529e1c590c411edbb1fdb974b25ae3dd1: Status 404 returned error can't find the container with id e9da76a3aa08694ff048ba92ae6614e529e1c590c411edbb1fdb974b25ae3dd1 Jul 27 20:41:29.755972 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:29.755926 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xv5xd" event={"ID":"0a117ca9-bd46-41c8-9b0c-5791ca8cb369","Type":"ContainerStarted","Data":"e9da76a3aa08694ff048ba92ae6614e529e1c590c411edbb1fdb974b25ae3dd1"} Jul 27 20:41:29.757629 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:29.757567 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-46drx" event={"ID":"92cbe4b8-7401-447f-a231-e7da3f656e94","Type":"ContainerStarted","Data":"36b1465c00bedd8a4cb610503b36f76c359885ad555c3eb8ad1f99704d42bd82"} Jul 27 20:41:31.771292 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:31.771254 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-46drx" event={"ID":"92cbe4b8-7401-447f-a231-e7da3f656e94","Type":"ContainerStarted","Data":"dc1984d6713d4a54682f4b0e0b89f572617d446744d35e964cf07944366afb27"} Jul 27 20:41:31.771819 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:31.771342 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-46drx" Jul 27 20:41:31.787401 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:31.787366 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk"] Jul 27 20:41:31.787584 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:31.787550 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk" podUID="0b672fab-a969-445e-89e4-b1668da529b8" containerName="discovery" containerID="cri-o://229417f3ef48c5d137ba787c5f499fb70cb9e8c02449315d3ee23c9ca5082560" gracePeriod=30 Jul 27 20:41:31.800168 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:31.800111 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-46drx" podStartSLOduration=1.975442699 podStartE2EDuration="4.800093796s" totalImagesPullingTime="2.824651097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:41:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:41:31.798584741 +0000 UTC m=+421.534565029" watchObservedRunningTime="2026-07-27 20:41:31.800093796 +0000 UTC m=+421.536074082" Jul 27 20:41:32.052388 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:32.052358 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk" Jul 27 20:41:32.114953 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:32.114919 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0b672fab-a969-445e-89e4-b1668da529b8-istio-csr-ca-configmap\") pod \"0b672fab-a969-445e-89e4-b1668da529b8\" (UID: \"0b672fab-a969-445e-89e4-b1668da529b8\") " Jul 27 20:41:32.115143 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:32.114965 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nt2gt\" (UniqueName: \"kubernetes.io/projected/0b672fab-a969-445e-89e4-b1668da529b8-kube-api-access-nt2gt\") pod \"0b672fab-a969-445e-89e4-b1668da529b8\" (UID: \"0b672fab-a969-445e-89e4-b1668da529b8\") " Jul 27 20:41:32.115143 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:32.114993 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0b672fab-a969-445e-89e4-b1668da529b8-cacerts\") pod \"0b672fab-a969-445e-89e4-b1668da529b8\" (UID: \"0b672fab-a969-445e-89e4-b1668da529b8\") " Jul 27 20:41:32.115143 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:32.115044 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b672fab-a969-445e-89e4-b1668da529b8-istio-token\") pod \"0b672fab-a969-445e-89e4-b1668da529b8\" (UID: \"0b672fab-a969-445e-89e4-b1668da529b8\") " Jul 27 20:41:32.115143 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:32.115089 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0b672fab-a969-445e-89e4-b1668da529b8-istio-csr-dns-cert\") pod \"0b672fab-a969-445e-89e4-b1668da529b8\" (UID: \"0b672fab-a969-445e-89e4-b1668da529b8\") " Jul 27 20:41:32.115143 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:32.115143 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0b672fab-a969-445e-89e4-b1668da529b8-istio-kubeconfig\") pod \"0b672fab-a969-445e-89e4-b1668da529b8\" (UID: \"0b672fab-a969-445e-89e4-b1668da529b8\") " Jul 27 20:41:32.115372 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:32.115206 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b672fab-a969-445e-89e4-b1668da529b8-local-certs\") pod \"0b672fab-a969-445e-89e4-b1668da529b8\" (UID: \"0b672fab-a969-445e-89e4-b1668da529b8\") " Jul 27 20:41:32.115429 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:32.115391 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b672fab-a969-445e-89e4-b1668da529b8-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "0b672fab-a969-445e-89e4-b1668da529b8" (UID: "0b672fab-a969-445e-89e4-b1668da529b8"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:41:32.115526 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:32.115511 2570 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0b672fab-a969-445e-89e4-b1668da529b8-istio-csr-ca-configmap\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:41:32.118294 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:32.118265 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b672fab-a969-445e-89e4-b1668da529b8-local-certs" (OuterVolumeSpecName: "local-certs") pod "0b672fab-a969-445e-89e4-b1668da529b8" (UID: "0b672fab-a969-445e-89e4-b1668da529b8"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 20:41:32.118469 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:32.118448 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b672fab-a969-445e-89e4-b1668da529b8-istio-token" (OuterVolumeSpecName: "istio-token") pod "0b672fab-a969-445e-89e4-b1668da529b8" (UID: "0b672fab-a969-445e-89e4-b1668da529b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:41:32.118642 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:32.118616 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b672fab-a969-445e-89e4-b1668da529b8-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "0b672fab-a969-445e-89e4-b1668da529b8" (UID: "0b672fab-a969-445e-89e4-b1668da529b8"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 20:41:32.118727 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:32.118695 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b672fab-a969-445e-89e4-b1668da529b8-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "0b672fab-a969-445e-89e4-b1668da529b8" (UID: "0b672fab-a969-445e-89e4-b1668da529b8"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 20:41:32.118806 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:32.118791 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b672fab-a969-445e-89e4-b1668da529b8-cacerts" (OuterVolumeSpecName: "cacerts") pod "0b672fab-a969-445e-89e4-b1668da529b8" (UID: "0b672fab-a969-445e-89e4-b1668da529b8"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 20:41:32.118850 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:32.118802 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b672fab-a969-445e-89e4-b1668da529b8-kube-api-access-nt2gt" (OuterVolumeSpecName: "kube-api-access-nt2gt") pod "0b672fab-a969-445e-89e4-b1668da529b8" (UID: "0b672fab-a969-445e-89e4-b1668da529b8"). InnerVolumeSpecName "kube-api-access-nt2gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:41:32.216493 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:32.216456 2570 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0b672fab-a969-445e-89e4-b1668da529b8-istio-kubeconfig\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:41:32.216493 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:32.216493 2570 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b672fab-a969-445e-89e4-b1668da529b8-local-certs\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:41:32.216753 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:32.216507 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt2gt\" (UniqueName: \"kubernetes.io/projected/0b672fab-a969-445e-89e4-b1668da529b8-kube-api-access-nt2gt\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:41:32.216753 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:32.216519 2570 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0b672fab-a969-445e-89e4-b1668da529b8-cacerts\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:41:32.216753 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:32.216530 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b672fab-a969-445e-89e4-b1668da529b8-istio-token\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:41:32.216753 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:32.216540 2570 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0b672fab-a969-445e-89e4-b1668da529b8-istio-csr-dns-cert\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:41:32.784899 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:32.784863 2570 generic.go:356] "Generic (PLEG): container finished" podID="0b672fab-a969-445e-89e4-b1668da529b8" containerID="229417f3ef48c5d137ba787c5f499fb70cb9e8c02449315d3ee23c9ca5082560" exitCode=0 Jul 27 20:41:32.785362 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:32.784937 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk" Jul 27 20:41:32.785362 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:32.784957 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk" event={"ID":"0b672fab-a969-445e-89e4-b1668da529b8","Type":"ContainerDied","Data":"229417f3ef48c5d137ba787c5f499fb70cb9e8c02449315d3ee23c9ca5082560"} Jul 27 20:41:32.785362 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:32.785063 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk" event={"ID":"0b672fab-a969-445e-89e4-b1668da529b8","Type":"ContainerDied","Data":"531474b63333968bf8f484c177b307ed13222850ff8a23b207bdd0ababc0dff7"} Jul 27 20:41:32.785362 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:32.785092 2570 scope.go:117] "RemoveContainer" containerID="229417f3ef48c5d137ba787c5f499fb70cb9e8c02449315d3ee23c9ca5082560" Jul 27 20:41:32.795344 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:32.795150 2570 scope.go:117] "RemoveContainer" containerID="229417f3ef48c5d137ba787c5f499fb70cb9e8c02449315d3ee23c9ca5082560" Jul 27 20:41:32.795778 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:41:32.795742 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"229417f3ef48c5d137ba787c5f499fb70cb9e8c02449315d3ee23c9ca5082560\": container with ID starting with 229417f3ef48c5d137ba787c5f499fb70cb9e8c02449315d3ee23c9ca5082560 not found: ID does not exist" containerID="229417f3ef48c5d137ba787c5f499fb70cb9e8c02449315d3ee23c9ca5082560" Jul 27 20:41:32.795890 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:32.795795 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"229417f3ef48c5d137ba787c5f499fb70cb9e8c02449315d3ee23c9ca5082560"} err="failed to get container status \"229417f3ef48c5d137ba787c5f499fb70cb9e8c02449315d3ee23c9ca5082560\": rpc error: code = NotFound desc = could not find container \"229417f3ef48c5d137ba787c5f499fb70cb9e8c02449315d3ee23c9ca5082560\": container with ID starting with 229417f3ef48c5d137ba787c5f499fb70cb9e8c02449315d3ee23c9ca5082560 not found: ID does not exist" Jul 27 20:41:32.809891 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:32.809859 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk"] Jul 27 20:41:32.815733 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:32.815705 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6psfk"] Jul 27 20:41:32.853096 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:32.853063 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b672fab-a969-445e-89e4-b1668da529b8" path="/var/lib/kubelet/pods/0b672fab-a969-445e-89e4-b1668da529b8/volumes" Jul 27 20:41:33.722706 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:33.722670 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-5vxxm" Jul 27 20:41:33.724664 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:33.724639 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-mssp2" Jul 27 20:41:33.792890 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:33.792847 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xv5xd" event={"ID":"0a117ca9-bd46-41c8-9b0c-5791ca8cb369","Type":"ContainerStarted","Data":"c36018e31104d6e690ed6e3efe3e066a1b162af3c46a78bc454b351eac359ea0"} Jul 27 20:41:33.793360 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:33.792954 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xv5xd" Jul 27 20:41:33.817744 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:41:33.817675 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xv5xd" podStartSLOduration=3.293862068 podStartE2EDuration="6.817655943s" totalImagesPullingTime="3.523793875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:41:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:41:33.815672838 +0000 UTC m=+423.551653124" watchObservedRunningTime="2026-07-27 20:41:33.817655943 +0000 UTC m=+423.553636229" Jul 27 20:42:02.790463 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:02.790380 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-46drx" Jul 27 20:42:04.803183 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:04.803150 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xv5xd" Jul 27 20:42:17.164656 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:17.164613 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bkjqh"] Jul 27 20:42:17.165211 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:17.165115 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b672fab-a969-445e-89e4-b1668da529b8" containerName="discovery" Jul 27 20:42:17.165211 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:17.165131 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b672fab-a969-445e-89e4-b1668da529b8" containerName="discovery" Jul 27 20:42:17.165351 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:17.165255 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b672fab-a969-445e-89e4-b1668da529b8" containerName="discovery" Jul 27 20:42:17.176304 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:17.176261 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-bkjqh" Jul 27 20:42:17.178699 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:17.178668 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bkjqh"] Jul 27 20:42:17.179076 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:17.179054 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 27 20:42:17.180069 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:17.180049 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-mjh8l\"" Jul 27 20:42:17.180316 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:17.180272 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 27 20:42:17.229287 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:17.229253 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj2xx\" (UniqueName: \"kubernetes.io/projected/7060638d-30df-4674-9973-1d9840f1da32-kube-api-access-dj2xx\") pod \"kuadrant-operator-catalog-bkjqh\" (UID: \"7060638d-30df-4674-9973-1d9840f1da32\") " pod="kuadrant-system/kuadrant-operator-catalog-bkjqh" Jul 27 20:42:17.330499 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:17.330463 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj2xx\" (UniqueName: \"kubernetes.io/projected/7060638d-30df-4674-9973-1d9840f1da32-kube-api-access-dj2xx\") pod \"kuadrant-operator-catalog-bkjqh\" (UID: \"7060638d-30df-4674-9973-1d9840f1da32\") " pod="kuadrant-system/kuadrant-operator-catalog-bkjqh" Jul 27 20:42:17.339076 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:17.339041 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj2xx\" (UniqueName: \"kubernetes.io/projected/7060638d-30df-4674-9973-1d9840f1da32-kube-api-access-dj2xx\") pod \"kuadrant-operator-catalog-bkjqh\" (UID: \"7060638d-30df-4674-9973-1d9840f1da32\") " pod="kuadrant-system/kuadrant-operator-catalog-bkjqh" Jul 27 20:42:17.488829 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:17.488786 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-bkjqh" Jul 27 20:42:17.628848 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:17.628820 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bkjqh"] Jul 27 20:42:17.630842 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:42:17.630813 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7060638d_30df_4674_9973_1d9840f1da32.slice/crio-cb64d073808d502efe922fa3f5d18658bbf2aaabfcf1059d9c0702a77dd3fd5b WatchSource:0}: Error finding container cb64d073808d502efe922fa3f5d18658bbf2aaabfcf1059d9c0702a77dd3fd5b: Status 404 returned error can't find the container with id cb64d073808d502efe922fa3f5d18658bbf2aaabfcf1059d9c0702a77dd3fd5b Jul 27 20:42:18.010592 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:18.010558 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-bkjqh" event={"ID":"7060638d-30df-4674-9973-1d9840f1da32","Type":"ContainerStarted","Data":"cb64d073808d502efe922fa3f5d18658bbf2aaabfcf1059d9c0702a77dd3fd5b"} Jul 27 20:42:20.023643 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:20.023582 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-bkjqh" event={"ID":"7060638d-30df-4674-9973-1d9840f1da32","Type":"ContainerStarted","Data":"673a33ac5ad2545d91508fe5159d7e559d60aea33111bc9a62de9f083d0c42ed"} Jul 27 20:42:20.037959 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:20.037906 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-bkjqh" podStartSLOduration=1.108858908 podStartE2EDuration="3.037890222s" totalImagesPullingTime="1.929031314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:42:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:42:20.037839765 +0000 UTC m=+469.773820052" watchObservedRunningTime="2026-07-27 20:42:20.037890222 +0000 UTC m=+469.773870508" Jul 27 20:42:27.489582 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:27.489541 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-bkjqh" Jul 27 20:42:27.490184 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:27.489626 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-bkjqh" Jul 27 20:42:27.512824 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:27.512798 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-bkjqh" Jul 27 20:42:28.083693 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:28.083667 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-bkjqh" Jul 27 20:42:31.382229 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:31.382195 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzk52"] Jul 27 20:42:31.387018 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:31.387000 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzk52" Jul 27 20:42:31.389453 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:31.389429 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6tzdk\"" Jul 27 20:42:31.393065 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:31.393039 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzk52"] Jul 27 20:42:31.458761 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:31.458720 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrxph\" (UniqueName: \"kubernetes.io/projected/9a32e2ac-8d7a-43de-afc2-3ebb48f3841a-kube-api-access-zrxph\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzk52\" (UID: \"9a32e2ac-8d7a-43de-afc2-3ebb48f3841a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzk52" Jul 27 20:42:31.458938 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:31.458776 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a32e2ac-8d7a-43de-afc2-3ebb48f3841a-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzk52\" (UID: \"9a32e2ac-8d7a-43de-afc2-3ebb48f3841a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzk52" Jul 27 20:42:31.458938 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:31.458822 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a32e2ac-8d7a-43de-afc2-3ebb48f3841a-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzk52\" (UID: \"9a32e2ac-8d7a-43de-afc2-3ebb48f3841a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzk52" Jul 27 20:42:31.560204 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:31.560166 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a32e2ac-8d7a-43de-afc2-3ebb48f3841a-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzk52\" (UID: \"9a32e2ac-8d7a-43de-afc2-3ebb48f3841a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzk52" Jul 27 20:42:31.560398 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:31.560248 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a32e2ac-8d7a-43de-afc2-3ebb48f3841a-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzk52\" (UID: \"9a32e2ac-8d7a-43de-afc2-3ebb48f3841a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzk52" Jul 27 20:42:31.560398 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:31.560324 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrxph\" (UniqueName: \"kubernetes.io/projected/9a32e2ac-8d7a-43de-afc2-3ebb48f3841a-kube-api-access-zrxph\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzk52\" (UID: \"9a32e2ac-8d7a-43de-afc2-3ebb48f3841a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzk52" Jul 27 20:42:31.560631 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:31.560582 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a32e2ac-8d7a-43de-afc2-3ebb48f3841a-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzk52\" (UID: \"9a32e2ac-8d7a-43de-afc2-3ebb48f3841a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzk52" Jul 27 20:42:31.560715 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:31.560653 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a32e2ac-8d7a-43de-afc2-3ebb48f3841a-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzk52\" (UID: \"9a32e2ac-8d7a-43de-afc2-3ebb48f3841a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzk52" Jul 27 20:42:31.569332 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:31.569298 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrxph\" (UniqueName: \"kubernetes.io/projected/9a32e2ac-8d7a-43de-afc2-3ebb48f3841a-kube-api-access-zrxph\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzk52\" (UID: \"9a32e2ac-8d7a-43de-afc2-3ebb48f3841a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzk52" Jul 27 20:42:31.697552 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:31.697463 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzk52" Jul 27 20:42:31.835813 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:31.835786 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzk52"] Jul 27 20:42:31.836625 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:42:31.836568 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a32e2ac_8d7a_43de_afc2_3ebb48f3841a.slice/crio-e6cf38c868d3a380e62b194bd7a68d8448aa1112ebd5ce87dce9ef592416a663 WatchSource:0}: Error finding container e6cf38c868d3a380e62b194bd7a68d8448aa1112ebd5ce87dce9ef592416a663: Status 404 returned error can't find the container with id e6cf38c868d3a380e62b194bd7a68d8448aa1112ebd5ce87dce9ef592416a663 Jul 27 20:42:31.981032 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:31.980997 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bzw4l"] Jul 27 20:42:31.984771 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:31.984748 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bzw4l" Jul 27 20:42:31.992230 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:31.992202 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bzw4l"] Jul 27 20:42:32.064381 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:32.064343 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ddf99cb9-8be9-4c04-ae1d-aee3ba69e320-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bzw4l\" (UID: \"ddf99cb9-8be9-4c04-ae1d-aee3ba69e320\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bzw4l" Jul 27 20:42:32.064567 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:32.064391 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ddf99cb9-8be9-4c04-ae1d-aee3ba69e320-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bzw4l\" (UID: \"ddf99cb9-8be9-4c04-ae1d-aee3ba69e320\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bzw4l" Jul 27 20:42:32.064567 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:32.064527 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjcpt\" (UniqueName: \"kubernetes.io/projected/ddf99cb9-8be9-4c04-ae1d-aee3ba69e320-kube-api-access-qjcpt\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bzw4l\" (UID: \"ddf99cb9-8be9-4c04-ae1d-aee3ba69e320\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bzw4l" Jul 27 20:42:32.086767 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:32.086725 2570 generic.go:356] "Generic (PLEG): container finished" podID="9a32e2ac-8d7a-43de-afc2-3ebb48f3841a" containerID="edfc00cebc7b5486929e4df1596374b10f34ddb2bc049cb1174a44b3ba54b4e2" exitCode=0 Jul 27 20:42:32.086926 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:32.086801 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzk52" event={"ID":"9a32e2ac-8d7a-43de-afc2-3ebb48f3841a","Type":"ContainerDied","Data":"edfc00cebc7b5486929e4df1596374b10f34ddb2bc049cb1174a44b3ba54b4e2"} Jul 27 20:42:32.086926 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:32.086850 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzk52" event={"ID":"9a32e2ac-8d7a-43de-afc2-3ebb48f3841a","Type":"ContainerStarted","Data":"e6cf38c868d3a380e62b194bd7a68d8448aa1112ebd5ce87dce9ef592416a663"} Jul 27 20:42:32.165940 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:32.165898 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ddf99cb9-8be9-4c04-ae1d-aee3ba69e320-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bzw4l\" (UID: \"ddf99cb9-8be9-4c04-ae1d-aee3ba69e320\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bzw4l" Jul 27 20:42:32.165940 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:32.165943 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ddf99cb9-8be9-4c04-ae1d-aee3ba69e320-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bzw4l\" (UID: \"ddf99cb9-8be9-4c04-ae1d-aee3ba69e320\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bzw4l" Jul 27 20:42:32.166160 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:32.166027 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjcpt\" (UniqueName: \"kubernetes.io/projected/ddf99cb9-8be9-4c04-ae1d-aee3ba69e320-kube-api-access-qjcpt\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bzw4l\" (UID: \"ddf99cb9-8be9-4c04-ae1d-aee3ba69e320\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bzw4l" Jul 27 20:42:32.166314 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:32.166293 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ddf99cb9-8be9-4c04-ae1d-aee3ba69e320-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bzw4l\" (UID: \"ddf99cb9-8be9-4c04-ae1d-aee3ba69e320\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bzw4l" Jul 27 20:42:32.166357 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:32.166338 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ddf99cb9-8be9-4c04-ae1d-aee3ba69e320-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bzw4l\" (UID: \"ddf99cb9-8be9-4c04-ae1d-aee3ba69e320\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bzw4l" Jul 27 20:42:32.176717 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:32.176685 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjcpt\" (UniqueName: \"kubernetes.io/projected/ddf99cb9-8be9-4c04-ae1d-aee3ba69e320-kube-api-access-qjcpt\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bzw4l\" (UID: \"ddf99cb9-8be9-4c04-ae1d-aee3ba69e320\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bzw4l" Jul 27 20:42:32.314794 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:32.314698 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bzw4l" Jul 27 20:42:32.453832 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:32.453805 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bzw4l"] Jul 27 20:42:32.456347 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:42:32.456314 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddf99cb9_8be9_4c04_ae1d_aee3ba69e320.slice/crio-c54b496b095ea73b3b71602cca2aff71ac862aa83e810635904f86b54c6ef336 WatchSource:0}: Error finding container c54b496b095ea73b3b71602cca2aff71ac862aa83e810635904f86b54c6ef336: Status 404 returned error can't find the container with id c54b496b095ea73b3b71602cca2aff71ac862aa83e810635904f86b54c6ef336 Jul 27 20:42:32.588193 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:32.588113 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pqhrm"] Jul 27 20:42:32.591847 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:32.591822 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pqhrm" Jul 27 20:42:32.601636 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:32.601593 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pqhrm"] Jul 27 20:42:32.671004 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:32.670966 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9231931-f505-4891-a372-c0581651da0b-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pqhrm\" (UID: \"d9231931-f505-4891-a372-c0581651da0b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pqhrm" Jul 27 20:42:32.671280 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:32.671078 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9231931-f505-4891-a372-c0581651da0b-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pqhrm\" (UID: \"d9231931-f505-4891-a372-c0581651da0b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pqhrm" Jul 27 20:42:32.671280 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:32.671105 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmdk8\" (UniqueName: \"kubernetes.io/projected/d9231931-f505-4891-a372-c0581651da0b-kube-api-access-nmdk8\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pqhrm\" (UID: \"d9231931-f505-4891-a372-c0581651da0b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pqhrm" Jul 27 20:42:32.772503 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:32.772467 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9231931-f505-4891-a372-c0581651da0b-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pqhrm\" (UID: \"d9231931-f505-4891-a372-c0581651da0b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pqhrm" Jul 27 20:42:32.772674 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:32.772552 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9231931-f505-4891-a372-c0581651da0b-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pqhrm\" (UID: \"d9231931-f505-4891-a372-c0581651da0b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pqhrm" Jul 27 20:42:32.772674 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:32.772580 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmdk8\" (UniqueName: \"kubernetes.io/projected/d9231931-f505-4891-a372-c0581651da0b-kube-api-access-nmdk8\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pqhrm\" (UID: \"d9231931-f505-4891-a372-c0581651da0b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pqhrm" Jul 27 20:42:32.772878 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:32.772859 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9231931-f505-4891-a372-c0581651da0b-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pqhrm\" (UID: \"d9231931-f505-4891-a372-c0581651da0b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pqhrm" Jul 27 20:42:32.772922 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:32.772894 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9231931-f505-4891-a372-c0581651da0b-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pqhrm\" (UID: \"d9231931-f505-4891-a372-c0581651da0b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pqhrm" Jul 27 20:42:32.781714 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:32.781690 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmdk8\" (UniqueName: \"kubernetes.io/projected/d9231931-f505-4891-a372-c0581651da0b-kube-api-access-nmdk8\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pqhrm\" (UID: \"d9231931-f505-4891-a372-c0581651da0b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pqhrm" Jul 27 20:42:32.933985 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:32.933951 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pqhrm" Jul 27 20:42:32.995356 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:32.995237 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t79wc"] Jul 27 20:42:33.007509 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:33.007480 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t79wc"] Jul 27 20:42:33.007830 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:33.007813 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t79wc" Jul 27 20:42:33.075307 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:33.075266 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwgj7\" (UniqueName: \"kubernetes.io/projected/3d05b9d7-e5bc-44c0-8e0d-67c1181d9340-kube-api-access-wwgj7\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t79wc\" (UID: \"3d05b9d7-e5bc-44c0-8e0d-67c1181d9340\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t79wc" Jul 27 20:42:33.075496 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:33.075322 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d05b9d7-e5bc-44c0-8e0d-67c1181d9340-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t79wc\" (UID: \"3d05b9d7-e5bc-44c0-8e0d-67c1181d9340\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t79wc" Jul 27 20:42:33.075496 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:33.075379 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d05b9d7-e5bc-44c0-8e0d-67c1181d9340-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t79wc\" (UID: \"3d05b9d7-e5bc-44c0-8e0d-67c1181d9340\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t79wc" Jul 27 20:42:33.093334 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:33.093299 2570 generic.go:356] "Generic (PLEG): container finished" podID="ddf99cb9-8be9-4c04-ae1d-aee3ba69e320" containerID="81943ed3eeb460aac93a12ad135993925dcdb072091e91c7dea462abecb0d76f" exitCode=0 Jul 27 20:42:33.093334 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:33.093343 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bzw4l" event={"ID":"ddf99cb9-8be9-4c04-ae1d-aee3ba69e320","Type":"ContainerDied","Data":"81943ed3eeb460aac93a12ad135993925dcdb072091e91c7dea462abecb0d76f"} Jul 27 20:42:33.093672 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:33.093391 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bzw4l" event={"ID":"ddf99cb9-8be9-4c04-ae1d-aee3ba69e320","Type":"ContainerStarted","Data":"c54b496b095ea73b3b71602cca2aff71ac862aa83e810635904f86b54c6ef336"} Jul 27 20:42:33.095434 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:33.095405 2570 generic.go:356] "Generic (PLEG): container finished" podID="9a32e2ac-8d7a-43de-afc2-3ebb48f3841a" containerID="11ca31b5c06c3e8d33d5669663567213a57e61f72f6c14c251ce508cc6ca09e5" exitCode=0 Jul 27 20:42:33.095543 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:33.095512 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzk52" event={"ID":"9a32e2ac-8d7a-43de-afc2-3ebb48f3841a","Type":"ContainerDied","Data":"11ca31b5c06c3e8d33d5669663567213a57e61f72f6c14c251ce508cc6ca09e5"} Jul 27 20:42:33.107492 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:33.107470 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pqhrm"] Jul 27 20:42:33.109343 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:42:33.109316 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9231931_f505_4891_a372_c0581651da0b.slice/crio-fa24d7e91c0761b34506b02cca3d3128f9379fe30ac709b6d997ae34491a6f53 WatchSource:0}: Error finding container fa24d7e91c0761b34506b02cca3d3128f9379fe30ac709b6d997ae34491a6f53: Status 404 returned error can't find the container with id fa24d7e91c0761b34506b02cca3d3128f9379fe30ac709b6d997ae34491a6f53 Jul 27 20:42:33.176722 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:33.176690 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d05b9d7-e5bc-44c0-8e0d-67c1181d9340-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t79wc\" (UID: \"3d05b9d7-e5bc-44c0-8e0d-67c1181d9340\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t79wc" Jul 27 20:42:33.176923 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:33.176804 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwgj7\" (UniqueName: \"kubernetes.io/projected/3d05b9d7-e5bc-44c0-8e0d-67c1181d9340-kube-api-access-wwgj7\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t79wc\" (UID: \"3d05b9d7-e5bc-44c0-8e0d-67c1181d9340\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t79wc" Jul 27 20:42:33.176923 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:33.176835 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d05b9d7-e5bc-44c0-8e0d-67c1181d9340-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t79wc\" (UID: \"3d05b9d7-e5bc-44c0-8e0d-67c1181d9340\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t79wc" Jul 27 20:42:33.177178 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:33.177138 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d05b9d7-e5bc-44c0-8e0d-67c1181d9340-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t79wc\" (UID: \"3d05b9d7-e5bc-44c0-8e0d-67c1181d9340\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t79wc" Jul 27 20:42:33.177302 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:33.177283 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d05b9d7-e5bc-44c0-8e0d-67c1181d9340-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t79wc\" (UID: \"3d05b9d7-e5bc-44c0-8e0d-67c1181d9340\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t79wc" Jul 27 20:42:33.185831 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:33.185792 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwgj7\" (UniqueName: \"kubernetes.io/projected/3d05b9d7-e5bc-44c0-8e0d-67c1181d9340-kube-api-access-wwgj7\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t79wc\" (UID: \"3d05b9d7-e5bc-44c0-8e0d-67c1181d9340\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t79wc" Jul 27 20:42:33.325443 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:33.325407 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t79wc" Jul 27 20:42:33.472790 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:33.472761 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t79wc"] Jul 27 20:42:33.474218 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:42:33.474188 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d05b9d7_e5bc_44c0_8e0d_67c1181d9340.slice/crio-7ac9f293fbd657568bc253b11d6bbe509fad8031617472a99262e9c24a663558 WatchSource:0}: Error finding container 7ac9f293fbd657568bc253b11d6bbe509fad8031617472a99262e9c24a663558: Status 404 returned error can't find the container with id 7ac9f293fbd657568bc253b11d6bbe509fad8031617472a99262e9c24a663558 Jul 27 20:42:34.102514 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:34.102424 2570 generic.go:356] "Generic (PLEG): container finished" podID="d9231931-f505-4891-a372-c0581651da0b" containerID="cc942777268d45549678e755b4c52af415f224ee6d6ddfc17a80676fd0ddc6a9" exitCode=0 Jul 27 20:42:34.102706 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:34.102517 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pqhrm" event={"ID":"d9231931-f505-4891-a372-c0581651da0b","Type":"ContainerDied","Data":"cc942777268d45549678e755b4c52af415f224ee6d6ddfc17a80676fd0ddc6a9"} Jul 27 20:42:34.102706 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:34.102627 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pqhrm" event={"ID":"d9231931-f505-4891-a372-c0581651da0b","Type":"ContainerStarted","Data":"fa24d7e91c0761b34506b02cca3d3128f9379fe30ac709b6d997ae34491a6f53"} Jul 27 20:42:34.104683 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:34.104655 2570 generic.go:356] "Generic (PLEG): container finished" podID="9a32e2ac-8d7a-43de-afc2-3ebb48f3841a" containerID="2a3e39fead34023a62a1eb6b354f1156078796635cfe9804bb1a9fceb45f1ffc" exitCode=0 Jul 27 20:42:34.104771 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:34.104752 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzk52" event={"ID":"9a32e2ac-8d7a-43de-afc2-3ebb48f3841a","Type":"ContainerDied","Data":"2a3e39fead34023a62a1eb6b354f1156078796635cfe9804bb1a9fceb45f1ffc"} Jul 27 20:42:34.106639 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:34.106619 2570 generic.go:356] "Generic (PLEG): container finished" podID="ddf99cb9-8be9-4c04-ae1d-aee3ba69e320" containerID="f979494c91cc75c21d023e5e35a54ee62a0809fea5871c35860a08fce4317b86" exitCode=0 Jul 27 20:42:34.106730 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:34.106697 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bzw4l" event={"ID":"ddf99cb9-8be9-4c04-ae1d-aee3ba69e320","Type":"ContainerDied","Data":"f979494c91cc75c21d023e5e35a54ee62a0809fea5871c35860a08fce4317b86"} Jul 27 20:42:34.108183 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:34.108166 2570 generic.go:356] "Generic (PLEG): container finished" podID="3d05b9d7-e5bc-44c0-8e0d-67c1181d9340" containerID="6935fce2f0110e559dd99b79f4537f8328b79e10d53012612db17cd677ca58f5" exitCode=0 Jul 27 20:42:34.108242 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:34.108198 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t79wc" event={"ID":"3d05b9d7-e5bc-44c0-8e0d-67c1181d9340","Type":"ContainerDied","Data":"6935fce2f0110e559dd99b79f4537f8328b79e10d53012612db17cd677ca58f5"} Jul 27 20:42:34.108242 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:34.108221 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t79wc" event={"ID":"3d05b9d7-e5bc-44c0-8e0d-67c1181d9340","Type":"ContainerStarted","Data":"7ac9f293fbd657568bc253b11d6bbe509fad8031617472a99262e9c24a663558"} Jul 27 20:42:34.267238 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:34.267194 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c9bc67bcd-7vfj5"] Jul 27 20:42:34.271218 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:34.271193 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c9bc67bcd-7vfj5" Jul 27 20:42:34.280841 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:34.280809 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c9bc67bcd-7vfj5"] Jul 27 20:42:34.389844 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:34.389755 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnrhm\" (UniqueName: \"kubernetes.io/projected/ed3a3437-420b-4fd4-bdc6-5fece371b252-kube-api-access-nnrhm\") pod \"console-5c9bc67bcd-7vfj5\" (UID: \"ed3a3437-420b-4fd4-bdc6-5fece371b252\") " pod="openshift-console/console-5c9bc67bcd-7vfj5" Jul 27 20:42:34.389844 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:34.389795 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed3a3437-420b-4fd4-bdc6-5fece371b252-console-serving-cert\") pod \"console-5c9bc67bcd-7vfj5\" (UID: \"ed3a3437-420b-4fd4-bdc6-5fece371b252\") " pod="openshift-console/console-5c9bc67bcd-7vfj5" Jul 27 20:42:34.390053 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:34.389896 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed3a3437-420b-4fd4-bdc6-5fece371b252-oauth-serving-cert\") pod \"console-5c9bc67bcd-7vfj5\" (UID: \"ed3a3437-420b-4fd4-bdc6-5fece371b252\") " pod="openshift-console/console-5c9bc67bcd-7vfj5" Jul 27 20:42:34.390053 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:34.389926 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed3a3437-420b-4fd4-bdc6-5fece371b252-service-ca\") pod \"console-5c9bc67bcd-7vfj5\" (UID: \"ed3a3437-420b-4fd4-bdc6-5fece371b252\") " pod="openshift-console/console-5c9bc67bcd-7vfj5" Jul 27 20:42:34.390053 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:34.389952 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed3a3437-420b-4fd4-bdc6-5fece371b252-console-oauth-config\") pod \"console-5c9bc67bcd-7vfj5\" (UID: \"ed3a3437-420b-4fd4-bdc6-5fece371b252\") " pod="openshift-console/console-5c9bc67bcd-7vfj5" Jul 27 20:42:34.390053 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:34.389993 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed3a3437-420b-4fd4-bdc6-5fece371b252-trusted-ca-bundle\") pod \"console-5c9bc67bcd-7vfj5\" (UID: \"ed3a3437-420b-4fd4-bdc6-5fece371b252\") " pod="openshift-console/console-5c9bc67bcd-7vfj5" Jul 27 20:42:34.390053 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:34.390011 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed3a3437-420b-4fd4-bdc6-5fece371b252-console-config\") pod \"console-5c9bc67bcd-7vfj5\" (UID: \"ed3a3437-420b-4fd4-bdc6-5fece371b252\") " pod="openshift-console/console-5c9bc67bcd-7vfj5" Jul 27 20:42:34.491110 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:34.491065 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnrhm\" (UniqueName: \"kubernetes.io/projected/ed3a3437-420b-4fd4-bdc6-5fece371b252-kube-api-access-nnrhm\") pod \"console-5c9bc67bcd-7vfj5\" (UID: \"ed3a3437-420b-4fd4-bdc6-5fece371b252\") " pod="openshift-console/console-5c9bc67bcd-7vfj5" Jul 27 20:42:34.491110 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:34.491112 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed3a3437-420b-4fd4-bdc6-5fece371b252-console-serving-cert\") pod \"console-5c9bc67bcd-7vfj5\" (UID: \"ed3a3437-420b-4fd4-bdc6-5fece371b252\") " pod="openshift-console/console-5c9bc67bcd-7vfj5" Jul 27 20:42:34.491633 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:34.491175 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed3a3437-420b-4fd4-bdc6-5fece371b252-oauth-serving-cert\") pod \"console-5c9bc67bcd-7vfj5\" (UID: \"ed3a3437-420b-4fd4-bdc6-5fece371b252\") " pod="openshift-console/console-5c9bc67bcd-7vfj5" Jul 27 20:42:34.491633 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:34.491193 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed3a3437-420b-4fd4-bdc6-5fece371b252-service-ca\") pod \"console-5c9bc67bcd-7vfj5\" (UID: \"ed3a3437-420b-4fd4-bdc6-5fece371b252\") " pod="openshift-console/console-5c9bc67bcd-7vfj5" Jul 27 20:42:34.491633 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:34.491221 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed3a3437-420b-4fd4-bdc6-5fece371b252-console-oauth-config\") pod \"console-5c9bc67bcd-7vfj5\" (UID: \"ed3a3437-420b-4fd4-bdc6-5fece371b252\") " pod="openshift-console/console-5c9bc67bcd-7vfj5" Jul 27 20:42:34.491633 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:34.491258 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed3a3437-420b-4fd4-bdc6-5fece371b252-trusted-ca-bundle\") pod \"console-5c9bc67bcd-7vfj5\" (UID: \"ed3a3437-420b-4fd4-bdc6-5fece371b252\") " pod="openshift-console/console-5c9bc67bcd-7vfj5" Jul 27 20:42:34.491633 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:34.491287 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed3a3437-420b-4fd4-bdc6-5fece371b252-console-config\") pod \"console-5c9bc67bcd-7vfj5\" (UID: \"ed3a3437-420b-4fd4-bdc6-5fece371b252\") " pod="openshift-console/console-5c9bc67bcd-7vfj5" Jul 27 20:42:34.492083 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:34.492059 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed3a3437-420b-4fd4-bdc6-5fece371b252-service-ca\") pod \"console-5c9bc67bcd-7vfj5\" (UID: \"ed3a3437-420b-4fd4-bdc6-5fece371b252\") " pod="openshift-console/console-5c9bc67bcd-7vfj5" Jul 27 20:42:34.492083 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:34.492074 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed3a3437-420b-4fd4-bdc6-5fece371b252-oauth-serving-cert\") pod \"console-5c9bc67bcd-7vfj5\" (UID: \"ed3a3437-420b-4fd4-bdc6-5fece371b252\") " pod="openshift-console/console-5c9bc67bcd-7vfj5" Jul 27 20:42:34.492194 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:34.492059 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed3a3437-420b-4fd4-bdc6-5fece371b252-console-config\") pod \"console-5c9bc67bcd-7vfj5\" (UID: \"ed3a3437-420b-4fd4-bdc6-5fece371b252\") " pod="openshift-console/console-5c9bc67bcd-7vfj5" Jul 27 20:42:34.492194 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:34.492167 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed3a3437-420b-4fd4-bdc6-5fece371b252-trusted-ca-bundle\") pod \"console-5c9bc67bcd-7vfj5\" (UID: \"ed3a3437-420b-4fd4-bdc6-5fece371b252\") " pod="openshift-console/console-5c9bc67bcd-7vfj5" Jul 27 20:42:34.494400 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:34.494367 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed3a3437-420b-4fd4-bdc6-5fece371b252-console-oauth-config\") pod \"console-5c9bc67bcd-7vfj5\" (UID: \"ed3a3437-420b-4fd4-bdc6-5fece371b252\") " pod="openshift-console/console-5c9bc67bcd-7vfj5" Jul 27 20:42:34.494504 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:34.494447 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed3a3437-420b-4fd4-bdc6-5fece371b252-console-serving-cert\") pod \"console-5c9bc67bcd-7vfj5\" (UID: \"ed3a3437-420b-4fd4-bdc6-5fece371b252\") " pod="openshift-console/console-5c9bc67bcd-7vfj5" Jul 27 20:42:34.500273 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:34.500249 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnrhm\" (UniqueName: \"kubernetes.io/projected/ed3a3437-420b-4fd4-bdc6-5fece371b252-kube-api-access-nnrhm\") pod \"console-5c9bc67bcd-7vfj5\" (UID: \"ed3a3437-420b-4fd4-bdc6-5fece371b252\") " pod="openshift-console/console-5c9bc67bcd-7vfj5" Jul 27 20:42:34.598068 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:34.597950 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c9bc67bcd-7vfj5" Jul 27 20:42:34.769524 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:34.769488 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c9bc67bcd-7vfj5"] Jul 27 20:42:34.824575 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:42:34.824549 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3a3437_420b_4fd4_bdc6_5fece371b252.slice/crio-806b74e90571da0d80e9a5f9e6ab56c96e05ed105d2833771247b2591d4c01c0 WatchSource:0}: Error finding container 806b74e90571da0d80e9a5f9e6ab56c96e05ed105d2833771247b2591d4c01c0: Status 404 returned error can't find the container with id 806b74e90571da0d80e9a5f9e6ab56c96e05ed105d2833771247b2591d4c01c0 Jul 27 20:42:35.116564 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:35.116530 2570 generic.go:356] "Generic (PLEG): container finished" podID="ddf99cb9-8be9-4c04-ae1d-aee3ba69e320" containerID="16afafe385e0c58f76b808c8bafddb0cc2fc4b4ca1278d1809d1b63b4e3f5908" exitCode=0 Jul 27 20:42:35.116785 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:35.116624 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bzw4l" event={"ID":"ddf99cb9-8be9-4c04-ae1d-aee3ba69e320","Type":"ContainerDied","Data":"16afafe385e0c58f76b808c8bafddb0cc2fc4b4ca1278d1809d1b63b4e3f5908"} Jul 27 20:42:35.118503 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:35.118472 2570 generic.go:356] "Generic (PLEG): container finished" podID="3d05b9d7-e5bc-44c0-8e0d-67c1181d9340" containerID="8195dcf0d92f0dc8324406309114059d4b2f2f6d3b20127cf9cb0c041ba3ca0f" exitCode=0 Jul 27 20:42:35.118665 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:35.118550 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t79wc" event={"ID":"3d05b9d7-e5bc-44c0-8e0d-67c1181d9340","Type":"ContainerDied","Data":"8195dcf0d92f0dc8324406309114059d4b2f2f6d3b20127cf9cb0c041ba3ca0f"} Jul 27 20:42:35.121063 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:35.121037 2570 generic.go:356] "Generic (PLEG): container finished" podID="d9231931-f505-4891-a372-c0581651da0b" containerID="c24ad8ead3322838737eb21cff831b629687713304ed22b7519267644ef7c3d2" exitCode=0 Jul 27 20:42:35.121168 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:35.121115 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pqhrm" event={"ID":"d9231931-f505-4891-a372-c0581651da0b","Type":"ContainerDied","Data":"c24ad8ead3322838737eb21cff831b629687713304ed22b7519267644ef7c3d2"} Jul 27 20:42:35.123061 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:35.123039 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c9bc67bcd-7vfj5" event={"ID":"ed3a3437-420b-4fd4-bdc6-5fece371b252","Type":"ContainerStarted","Data":"546385acf30af0fcb781266b465ec9de9b6f1013efd803b3c1b9bff76e8100f0"} Jul 27 20:42:35.123154 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:35.123080 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c9bc67bcd-7vfj5" event={"ID":"ed3a3437-420b-4fd4-bdc6-5fece371b252","Type":"ContainerStarted","Data":"806b74e90571da0d80e9a5f9e6ab56c96e05ed105d2833771247b2591d4c01c0"} Jul 27 20:42:35.172506 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:35.172447 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c9bc67bcd-7vfj5" podStartSLOduration=1.17242503 podStartE2EDuration="1.17242503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:42:35.169379313 +0000 UTC m=+484.905359601" watchObservedRunningTime="2026-07-27 20:42:35.17242503 +0000 UTC m=+484.908405318" Jul 27 20:42:35.269138 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:35.269104 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzk52" Jul 27 20:42:35.401000 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:35.400957 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a32e2ac-8d7a-43de-afc2-3ebb48f3841a-util\") pod \"9a32e2ac-8d7a-43de-afc2-3ebb48f3841a\" (UID: \"9a32e2ac-8d7a-43de-afc2-3ebb48f3841a\") " Jul 27 20:42:35.401171 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:35.401024 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a32e2ac-8d7a-43de-afc2-3ebb48f3841a-bundle\") pod \"9a32e2ac-8d7a-43de-afc2-3ebb48f3841a\" (UID: \"9a32e2ac-8d7a-43de-afc2-3ebb48f3841a\") " Jul 27 20:42:35.401171 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:35.401070 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrxph\" (UniqueName: \"kubernetes.io/projected/9a32e2ac-8d7a-43de-afc2-3ebb48f3841a-kube-api-access-zrxph\") pod \"9a32e2ac-8d7a-43de-afc2-3ebb48f3841a\" (UID: \"9a32e2ac-8d7a-43de-afc2-3ebb48f3841a\") " Jul 27 20:42:35.401528 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:35.401504 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a32e2ac-8d7a-43de-afc2-3ebb48f3841a-bundle" (OuterVolumeSpecName: "bundle") pod "9a32e2ac-8d7a-43de-afc2-3ebb48f3841a" (UID: "9a32e2ac-8d7a-43de-afc2-3ebb48f3841a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 20:42:35.403416 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:35.403392 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a32e2ac-8d7a-43de-afc2-3ebb48f3841a-kube-api-access-zrxph" (OuterVolumeSpecName: "kube-api-access-zrxph") pod "9a32e2ac-8d7a-43de-afc2-3ebb48f3841a" (UID: "9a32e2ac-8d7a-43de-afc2-3ebb48f3841a"). InnerVolumeSpecName "kube-api-access-zrxph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:42:35.406389 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:35.406362 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a32e2ac-8d7a-43de-afc2-3ebb48f3841a-util" (OuterVolumeSpecName: "util") pod "9a32e2ac-8d7a-43de-afc2-3ebb48f3841a" (UID: "9a32e2ac-8d7a-43de-afc2-3ebb48f3841a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 20:42:35.501953 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:35.501912 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrxph\" (UniqueName: \"kubernetes.io/projected/9a32e2ac-8d7a-43de-afc2-3ebb48f3841a-kube-api-access-zrxph\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:42:35.501953 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:35.501949 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a32e2ac-8d7a-43de-afc2-3ebb48f3841a-util\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:42:35.501953 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:35.501961 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a32e2ac-8d7a-43de-afc2-3ebb48f3841a-bundle\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:42:36.130719 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:36.130686 2570 generic.go:356] "Generic (PLEG): container finished" podID="3d05b9d7-e5bc-44c0-8e0d-67c1181d9340" containerID="138afef2e44acd15139a30cbb1773044aef58fa1db61762713ae43c8d37c1ea8" exitCode=0 Jul 27 20:42:36.130868 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:36.130779 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t79wc" event={"ID":"3d05b9d7-e5bc-44c0-8e0d-67c1181d9340","Type":"ContainerDied","Data":"138afef2e44acd15139a30cbb1773044aef58fa1db61762713ae43c8d37c1ea8"} Jul 27 20:42:36.132804 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:36.132775 2570 generic.go:356] "Generic (PLEG): container finished" podID="d9231931-f505-4891-a372-c0581651da0b" containerID="094559dcafc0b3765c47382fd121b188b041fb3e099659c66602d79c0149fd65" exitCode=0 Jul 27 20:42:36.132927 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:36.132850 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pqhrm" event={"ID":"d9231931-f505-4891-a372-c0581651da0b","Type":"ContainerDied","Data":"094559dcafc0b3765c47382fd121b188b041fb3e099659c66602d79c0149fd65"} Jul 27 20:42:36.134901 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:36.134884 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzk52" Jul 27 20:42:36.135001 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:36.134901 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzk52" event={"ID":"9a32e2ac-8d7a-43de-afc2-3ebb48f3841a","Type":"ContainerDied","Data":"e6cf38c868d3a380e62b194bd7a68d8448aa1112ebd5ce87dce9ef592416a663"} Jul 27 20:42:36.135001 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:36.134937 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6cf38c868d3a380e62b194bd7a68d8448aa1112ebd5ce87dce9ef592416a663" Jul 27 20:42:36.273322 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:36.273297 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bzw4l" Jul 27 20:42:36.411057 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:36.410953 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ddf99cb9-8be9-4c04-ae1d-aee3ba69e320-bundle\") pod \"ddf99cb9-8be9-4c04-ae1d-aee3ba69e320\" (UID: \"ddf99cb9-8be9-4c04-ae1d-aee3ba69e320\") " Jul 27 20:42:36.411057 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:36.410999 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ddf99cb9-8be9-4c04-ae1d-aee3ba69e320-util\") pod \"ddf99cb9-8be9-4c04-ae1d-aee3ba69e320\" (UID: \"ddf99cb9-8be9-4c04-ae1d-aee3ba69e320\") " Jul 27 20:42:36.411278 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:36.411070 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjcpt\" (UniqueName: \"kubernetes.io/projected/ddf99cb9-8be9-4c04-ae1d-aee3ba69e320-kube-api-access-qjcpt\") pod \"ddf99cb9-8be9-4c04-ae1d-aee3ba69e320\" (UID: \"ddf99cb9-8be9-4c04-ae1d-aee3ba69e320\") " Jul 27 20:42:36.411470 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:36.411442 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddf99cb9-8be9-4c04-ae1d-aee3ba69e320-bundle" (OuterVolumeSpecName: "bundle") pod "ddf99cb9-8be9-4c04-ae1d-aee3ba69e320" (UID: "ddf99cb9-8be9-4c04-ae1d-aee3ba69e320"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 20:42:36.413302 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:36.413280 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddf99cb9-8be9-4c04-ae1d-aee3ba69e320-kube-api-access-qjcpt" (OuterVolumeSpecName: "kube-api-access-qjcpt") pod "ddf99cb9-8be9-4c04-ae1d-aee3ba69e320" (UID: "ddf99cb9-8be9-4c04-ae1d-aee3ba69e320"). InnerVolumeSpecName "kube-api-access-qjcpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:42:36.416723 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:36.416687 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddf99cb9-8be9-4c04-ae1d-aee3ba69e320-util" (OuterVolumeSpecName: "util") pod "ddf99cb9-8be9-4c04-ae1d-aee3ba69e320" (UID: "ddf99cb9-8be9-4c04-ae1d-aee3ba69e320"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 20:42:36.512469 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:36.512429 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjcpt\" (UniqueName: \"kubernetes.io/projected/ddf99cb9-8be9-4c04-ae1d-aee3ba69e320-kube-api-access-qjcpt\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:42:36.512469 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:36.512462 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ddf99cb9-8be9-4c04-ae1d-aee3ba69e320-bundle\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:42:36.512469 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:36.512471 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ddf99cb9-8be9-4c04-ae1d-aee3ba69e320-util\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:42:37.142378 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:37.142291 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bzw4l" Jul 27 20:42:37.142378 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:37.142308 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bzw4l" event={"ID":"ddf99cb9-8be9-4c04-ae1d-aee3ba69e320","Type":"ContainerDied","Data":"c54b496b095ea73b3b71602cca2aff71ac862aa83e810635904f86b54c6ef336"} Jul 27 20:42:37.142378 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:37.142360 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c54b496b095ea73b3b71602cca2aff71ac862aa83e810635904f86b54c6ef336" Jul 27 20:42:37.303853 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:37.303827 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t79wc" Jul 27 20:42:37.305894 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:37.305872 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pqhrm" Jul 27 20:42:37.420425 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:37.420328 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9231931-f505-4891-a372-c0581651da0b-util\") pod \"d9231931-f505-4891-a372-c0581651da0b\" (UID: \"d9231931-f505-4891-a372-c0581651da0b\") " Jul 27 20:42:37.420425 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:37.420386 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d05b9d7-e5bc-44c0-8e0d-67c1181d9340-bundle\") pod \"3d05b9d7-e5bc-44c0-8e0d-67c1181d9340\" (UID: \"3d05b9d7-e5bc-44c0-8e0d-67c1181d9340\") " Jul 27 20:42:37.420652 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:37.420430 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d05b9d7-e5bc-44c0-8e0d-67c1181d9340-util\") pod \"3d05b9d7-e5bc-44c0-8e0d-67c1181d9340\" (UID: \"3d05b9d7-e5bc-44c0-8e0d-67c1181d9340\") " Jul 27 20:42:37.420652 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:37.420496 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwgj7\" (UniqueName: \"kubernetes.io/projected/3d05b9d7-e5bc-44c0-8e0d-67c1181d9340-kube-api-access-wwgj7\") pod \"3d05b9d7-e5bc-44c0-8e0d-67c1181d9340\" (UID: \"3d05b9d7-e5bc-44c0-8e0d-67c1181d9340\") " Jul 27 20:42:37.420652 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:37.420521 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9231931-f505-4891-a372-c0581651da0b-bundle\") pod \"d9231931-f505-4891-a372-c0581651da0b\" (UID: \"d9231931-f505-4891-a372-c0581651da0b\") " Jul 27 20:42:37.420652 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:37.420555 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmdk8\" (UniqueName: \"kubernetes.io/projected/d9231931-f505-4891-a372-c0581651da0b-kube-api-access-nmdk8\") pod \"d9231931-f505-4891-a372-c0581651da0b\" (UID: \"d9231931-f505-4891-a372-c0581651da0b\") " Jul 27 20:42:37.421091 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:37.421056 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d05b9d7-e5bc-44c0-8e0d-67c1181d9340-bundle" (OuterVolumeSpecName: "bundle") pod "3d05b9d7-e5bc-44c0-8e0d-67c1181d9340" (UID: "3d05b9d7-e5bc-44c0-8e0d-67c1181d9340"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 20:42:37.421500 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:37.421469 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9231931-f505-4891-a372-c0581651da0b-bundle" (OuterVolumeSpecName: "bundle") pod "d9231931-f505-4891-a372-c0581651da0b" (UID: "d9231931-f505-4891-a372-c0581651da0b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 20:42:37.422967 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:37.422944 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d05b9d7-e5bc-44c0-8e0d-67c1181d9340-kube-api-access-wwgj7" (OuterVolumeSpecName: "kube-api-access-wwgj7") pod "3d05b9d7-e5bc-44c0-8e0d-67c1181d9340" (UID: "3d05b9d7-e5bc-44c0-8e0d-67c1181d9340"). InnerVolumeSpecName "kube-api-access-wwgj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:42:37.423039 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:37.422985 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9231931-f505-4891-a372-c0581651da0b-kube-api-access-nmdk8" (OuterVolumeSpecName: "kube-api-access-nmdk8") pod "d9231931-f505-4891-a372-c0581651da0b" (UID: "d9231931-f505-4891-a372-c0581651da0b"). InnerVolumeSpecName "kube-api-access-nmdk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:42:37.426121 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:37.426100 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d05b9d7-e5bc-44c0-8e0d-67c1181d9340-util" (OuterVolumeSpecName: "util") pod "3d05b9d7-e5bc-44c0-8e0d-67c1181d9340" (UID: "3d05b9d7-e5bc-44c0-8e0d-67c1181d9340"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 20:42:37.427288 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:37.427265 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9231931-f505-4891-a372-c0581651da0b-util" (OuterVolumeSpecName: "util") pod "d9231931-f505-4891-a372-c0581651da0b" (UID: "d9231931-f505-4891-a372-c0581651da0b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 20:42:37.522188 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:37.522153 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwgj7\" (UniqueName: \"kubernetes.io/projected/3d05b9d7-e5bc-44c0-8e0d-67c1181d9340-kube-api-access-wwgj7\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:42:37.522188 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:37.522182 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9231931-f505-4891-a372-c0581651da0b-bundle\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:42:37.522188 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:37.522191 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmdk8\" (UniqueName: \"kubernetes.io/projected/d9231931-f505-4891-a372-c0581651da0b-kube-api-access-nmdk8\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:42:37.522683 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:37.522201 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9231931-f505-4891-a372-c0581651da0b-util\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:42:37.522683 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:37.522210 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d05b9d7-e5bc-44c0-8e0d-67c1181d9340-bundle\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:42:37.522683 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:37.522218 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d05b9d7-e5bc-44c0-8e0d-67c1181d9340-util\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:42:38.149128 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:38.149097 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pqhrm" Jul 27 20:42:38.149128 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:38.149121 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pqhrm" event={"ID":"d9231931-f505-4891-a372-c0581651da0b","Type":"ContainerDied","Data":"fa24d7e91c0761b34506b02cca3d3128f9379fe30ac709b6d997ae34491a6f53"} Jul 27 20:42:38.149380 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:38.149168 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa24d7e91c0761b34506b02cca3d3128f9379fe30ac709b6d997ae34491a6f53" Jul 27 20:42:38.150996 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:38.150972 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t79wc" event={"ID":"3d05b9d7-e5bc-44c0-8e0d-67c1181d9340","Type":"ContainerDied","Data":"7ac9f293fbd657568bc253b11d6bbe509fad8031617472a99262e9c24a663558"} Jul 27 20:42:38.151121 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:38.151008 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ac9f293fbd657568bc253b11d6bbe509fad8031617472a99262e9c24a663558" Jul 27 20:42:38.151121 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:38.151016 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t79wc" Jul 27 20:42:44.598930 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:44.598890 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c9bc67bcd-7vfj5" Jul 27 20:42:44.599310 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:44.598993 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c9bc67bcd-7vfj5" Jul 27 20:42:44.604062 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:44.604039 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c9bc67bcd-7vfj5" Jul 27 20:42:45.189491 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:45.189462 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c9bc67bcd-7vfj5" Jul 27 20:42:45.254121 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:45.254084 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c484c9c79-ttn8p"] Jul 27 20:42:48.413673 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.413637 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-5xm4m"] Jul 27 20:42:48.414057 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.414000 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddf99cb9-8be9-4c04-ae1d-aee3ba69e320" containerName="extract" Jul 27 20:42:48.414057 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.414011 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddf99cb9-8be9-4c04-ae1d-aee3ba69e320" containerName="extract" Jul 27 20:42:48.414057 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.414020 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a32e2ac-8d7a-43de-afc2-3ebb48f3841a" containerName="pull" Jul 27 20:42:48.414057 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.414026 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a32e2ac-8d7a-43de-afc2-3ebb48f3841a" containerName="pull" Jul 27 20:42:48.414057 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.414037 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9231931-f505-4891-a372-c0581651da0b" containerName="util" Jul 27 20:42:48.414057 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.414046 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9231931-f505-4891-a372-c0581651da0b" containerName="util" Jul 27 20:42:48.414057 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.414058 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddf99cb9-8be9-4c04-ae1d-aee3ba69e320" containerName="util" Jul 27 20:42:48.414257 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.414066 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddf99cb9-8be9-4c04-ae1d-aee3ba69e320" containerName="util" Jul 27 20:42:48.414257 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.414075 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d05b9d7-e5bc-44c0-8e0d-67c1181d9340" containerName="pull" Jul 27 20:42:48.414257 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.414080 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d05b9d7-e5bc-44c0-8e0d-67c1181d9340" containerName="pull" Jul 27 20:42:48.414257 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.414091 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a32e2ac-8d7a-43de-afc2-3ebb48f3841a" containerName="util" Jul 27 20:42:48.414257 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.414097 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a32e2ac-8d7a-43de-afc2-3ebb48f3841a" containerName="util" Jul 27 20:42:48.414257 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.414102 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d05b9d7-e5bc-44c0-8e0d-67c1181d9340" containerName="util" Jul 27 20:42:48.414257 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.414107 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d05b9d7-e5bc-44c0-8e0d-67c1181d9340" containerName="util" Jul 27 20:42:48.414257 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.414112 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d05b9d7-e5bc-44c0-8e0d-67c1181d9340" containerName="extract" Jul 27 20:42:48.414257 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.414117 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d05b9d7-e5bc-44c0-8e0d-67c1181d9340" containerName="extract" Jul 27 20:42:48.414257 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.414125 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddf99cb9-8be9-4c04-ae1d-aee3ba69e320" containerName="pull" Jul 27 20:42:48.414257 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.414130 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddf99cb9-8be9-4c04-ae1d-aee3ba69e320" containerName="pull" Jul 27 20:42:48.414257 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.414137 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9231931-f505-4891-a372-c0581651da0b" containerName="extract" Jul 27 20:42:48.414257 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.414141 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9231931-f505-4891-a372-c0581651da0b" containerName="extract" Jul 27 20:42:48.414257 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.414148 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9231931-f505-4891-a372-c0581651da0b" containerName="pull" Jul 27 20:42:48.414257 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.414152 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9231931-f505-4891-a372-c0581651da0b" containerName="pull" Jul 27 20:42:48.414257 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.414158 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a32e2ac-8d7a-43de-afc2-3ebb48f3841a" containerName="extract" Jul 27 20:42:48.414257 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.414163 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a32e2ac-8d7a-43de-afc2-3ebb48f3841a" containerName="extract" Jul 27 20:42:48.414257 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.414217 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d05b9d7-e5bc-44c0-8e0d-67c1181d9340" containerName="extract" Jul 27 20:42:48.414257 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.414225 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a32e2ac-8d7a-43de-afc2-3ebb48f3841a" containerName="extract" Jul 27 20:42:48.414257 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.414231 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddf99cb9-8be9-4c04-ae1d-aee3ba69e320" containerName="extract" Jul 27 20:42:48.414257 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.414237 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9231931-f505-4891-a372-c0581651da0b" containerName="extract" Jul 27 20:42:48.418481 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.418460 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5xm4m" Jul 27 20:42:48.424575 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.424550 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-plrmz\"" Jul 27 20:42:48.431309 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.431285 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-5xm4m"] Jul 27 20:42:48.521023 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.520981 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-277xc\" (UniqueName: \"kubernetes.io/projected/8dc05f60-181a-4b57-8e9f-547c6b773d40-kube-api-access-277xc\") pod \"authorino-operator-75cc8d5cb7-5xm4m\" (UID: \"8dc05f60-181a-4b57-8e9f-547c6b773d40\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-5xm4m" Jul 27 20:42:48.622033 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.621964 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-277xc\" (UniqueName: \"kubernetes.io/projected/8dc05f60-181a-4b57-8e9f-547c6b773d40-kube-api-access-277xc\") pod \"authorino-operator-75cc8d5cb7-5xm4m\" (UID: \"8dc05f60-181a-4b57-8e9f-547c6b773d40\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-5xm4m" Jul 27 20:42:48.633380 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.633353 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-277xc\" (UniqueName: \"kubernetes.io/projected/8dc05f60-181a-4b57-8e9f-547c6b773d40-kube-api-access-277xc\") pod \"authorino-operator-75cc8d5cb7-5xm4m\" (UID: \"8dc05f60-181a-4b57-8e9f-547c6b773d40\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-5xm4m" Jul 27 20:42:48.729023 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.728973 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5xm4m" Jul 27 20:42:48.871052 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:48.871026 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-5xm4m"] Jul 27 20:42:48.872709 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:42:48.872679 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dc05f60_181a_4b57_8e9f_547c6b773d40.slice/crio-cb043afc56598f72a2b51a59c47c217610c4b51bc14829b06fd7ee8887f694dc WatchSource:0}: Error finding container cb043afc56598f72a2b51a59c47c217610c4b51bc14829b06fd7ee8887f694dc: Status 404 returned error can't find the container with id cb043afc56598f72a2b51a59c47c217610c4b51bc14829b06fd7ee8887f694dc Jul 27 20:42:49.214453 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:49.214412 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5xm4m" event={"ID":"8dc05f60-181a-4b57-8e9f-547c6b773d40","Type":"ContainerStarted","Data":"cb043afc56598f72a2b51a59c47c217610c4b51bc14829b06fd7ee8887f694dc"} Jul 27 20:42:51.227055 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:51.227019 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5xm4m" event={"ID":"8dc05f60-181a-4b57-8e9f-547c6b773d40","Type":"ContainerStarted","Data":"a9fdec0d8fc2406b6e0a6842617ac32363aae07c9bb07dcc14107f4c201ad834"} Jul 27 20:42:51.227430 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:51.227093 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5xm4m" Jul 27 20:42:51.246363 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:51.246311 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5xm4m" podStartSLOduration=1.180324338 podStartE2EDuration="3.246295255s" totalImagesPullingTime="2.065970917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:42:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:42:51.244161281 +0000 UTC m=+500.980141597" watchObservedRunningTime="2026-07-27 20:42:51.246295255 +0000 UTC m=+500.982275542" Jul 27 20:42:51.648918 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:51.648833 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf8pq"] Jul 27 20:42:51.652527 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:51.651661 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf8pq" Jul 27 20:42:51.660268 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:51.660239 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-9bl7g\"" Jul 27 20:42:51.666661 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:51.666632 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf8pq"] Jul 27 20:42:51.748911 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:51.748873 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr8vx\" (UniqueName: \"kubernetes.io/projected/82f831ea-3672-4636-be8a-d50cef83fa45-kube-api-access-sr8vx\") pod \"limitador-operator-controller-manager-859d7b78bd-zf8pq\" (UID: \"82f831ea-3672-4636-be8a-d50cef83fa45\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf8pq" Jul 27 20:42:51.850140 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:51.850083 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr8vx\" (UniqueName: \"kubernetes.io/projected/82f831ea-3672-4636-be8a-d50cef83fa45-kube-api-access-sr8vx\") pod \"limitador-operator-controller-manager-859d7b78bd-zf8pq\" (UID: \"82f831ea-3672-4636-be8a-d50cef83fa45\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf8pq" Jul 27 20:42:51.866951 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:51.866923 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr8vx\" (UniqueName: \"kubernetes.io/projected/82f831ea-3672-4636-be8a-d50cef83fa45-kube-api-access-sr8vx\") pod \"limitador-operator-controller-manager-859d7b78bd-zf8pq\" (UID: \"82f831ea-3672-4636-be8a-d50cef83fa45\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf8pq" Jul 27 20:42:51.972098 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:51.972064 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf8pq" Jul 27 20:42:52.106669 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:52.106642 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf8pq"] Jul 27 20:42:52.107953 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:42:52.107930 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f831ea_3672_4636_be8a_d50cef83fa45.slice/crio-7e4dcbc6d3c32aa33a7c21a2cef7c9875cf9ef1150504de7a721b17952032399 WatchSource:0}: Error finding container 7e4dcbc6d3c32aa33a7c21a2cef7c9875cf9ef1150504de7a721b17952032399: Status 404 returned error can't find the container with id 7e4dcbc6d3c32aa33a7c21a2cef7c9875cf9ef1150504de7a721b17952032399 Jul 27 20:42:52.234075 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:52.233989 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf8pq" event={"ID":"82f831ea-3672-4636-be8a-d50cef83fa45","Type":"ContainerStarted","Data":"7e4dcbc6d3c32aa33a7c21a2cef7c9875cf9ef1150504de7a721b17952032399"} Jul 27 20:42:54.246122 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:54.246075 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf8pq" event={"ID":"82f831ea-3672-4636-be8a-d50cef83fa45","Type":"ContainerStarted","Data":"5231b340d22c541d0491c40c84ce542d9d4bf7ef6d13473e12e5eb293c526237"} Jul 27 20:42:54.246515 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:54.246157 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf8pq" Jul 27 20:42:54.265790 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:54.265732 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf8pq" podStartSLOduration=1.485736052 podStartE2EDuration="3.265716305s" totalImagesPullingTime="1.779980253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:42:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:42:54.263112756 +0000 UTC m=+503.999093052" watchObservedRunningTime="2026-07-27 20:42:54.265716305 +0000 UTC m=+504.001696592" Jul 27 20:42:54.973253 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:54.973214 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-znhw2"] Jul 27 20:42:54.976946 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:54.976926 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-znhw2" Jul 27 20:42:54.980186 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:54.980161 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-fv6f8\"" Jul 27 20:42:54.980403 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:54.980382 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 27 20:42:54.990387 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:54.990360 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-znhw2"] Jul 27 20:42:55.083211 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:55.083175 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzz76\" (UniqueName: \"kubernetes.io/projected/b6f5fe4c-b4c4-4deb-a0ee-c7d7a9ad38f2-kube-api-access-bzz76\") pod \"dns-operator-controller-manager-864dd78c4c-znhw2\" (UID: \"b6f5fe4c-b4c4-4deb-a0ee-c7d7a9ad38f2\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-znhw2" Jul 27 20:42:55.184241 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:55.184194 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzz76\" (UniqueName: \"kubernetes.io/projected/b6f5fe4c-b4c4-4deb-a0ee-c7d7a9ad38f2-kube-api-access-bzz76\") pod \"dns-operator-controller-manager-864dd78c4c-znhw2\" (UID: \"b6f5fe4c-b4c4-4deb-a0ee-c7d7a9ad38f2\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-znhw2" Jul 27 20:42:55.193870 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:55.193837 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzz76\" (UniqueName: \"kubernetes.io/projected/b6f5fe4c-b4c4-4deb-a0ee-c7d7a9ad38f2-kube-api-access-bzz76\") pod \"dns-operator-controller-manager-864dd78c4c-znhw2\" (UID: \"b6f5fe4c-b4c4-4deb-a0ee-c7d7a9ad38f2\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-znhw2" Jul 27 20:42:55.287978 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:55.287883 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-znhw2" Jul 27 20:42:55.433709 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:55.433679 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-znhw2"] Jul 27 20:42:55.435277 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:42:55.435250 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6f5fe4c_b4c4_4deb_a0ee_c7d7a9ad38f2.slice/crio-d3f36a26708efbbbdb8eb24efe1cfe1db29e8636cc4a66279f63d0ca23ba80a7 WatchSource:0}: Error finding container d3f36a26708efbbbdb8eb24efe1cfe1db29e8636cc4a66279f63d0ca23ba80a7: Status 404 returned error can't find the container with id d3f36a26708efbbbdb8eb24efe1cfe1db29e8636cc4a66279f63d0ca23ba80a7 Jul 27 20:42:56.259802 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:56.259770 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-znhw2" event={"ID":"b6f5fe4c-b4c4-4deb-a0ee-c7d7a9ad38f2","Type":"ContainerStarted","Data":"d3f36a26708efbbbdb8eb24efe1cfe1db29e8636cc4a66279f63d0ca23ba80a7"} Jul 27 20:42:58.273514 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:58.273476 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-znhw2" event={"ID":"b6f5fe4c-b4c4-4deb-a0ee-c7d7a9ad38f2","Type":"ContainerStarted","Data":"bd93059884efdd6b4cd1b045c0624150378fe14098278816c74b39c1e4d7d5b3"} Jul 27 20:42:58.273918 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:58.273610 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-znhw2" Jul 27 20:42:58.291186 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:42:58.291131 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-znhw2" podStartSLOduration=2.225452993 podStartE2EDuration="4.291116021s" totalImagesPullingTime="2.065663028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:42:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:42:58.289327327 +0000 UTC m=+508.025307614" watchObservedRunningTime="2026-07-27 20:42:58.291116021 +0000 UTC m=+508.027096349" Jul 27 20:43:00.806144 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:00.806108 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bslzn"] Jul 27 20:43:00.809791 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:00.809771 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bslzn" Jul 27 20:43:00.812642 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:00.812587 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-2nshg\"" Jul 27 20:43:00.825037 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:00.825011 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bslzn"] Jul 27 20:43:00.840114 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:00.840080 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgc8p\" (UniqueName: \"kubernetes.io/projected/4c2f6be6-ed7c-44c1-9203-35245a64cf0d-kube-api-access-sgc8p\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bslzn\" (UID: \"4c2f6be6-ed7c-44c1-9203-35245a64cf0d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bslzn" Jul 27 20:43:00.840293 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:00.840181 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c2f6be6-ed7c-44c1-9203-35245a64cf0d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bslzn\" (UID: \"4c2f6be6-ed7c-44c1-9203-35245a64cf0d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bslzn" Jul 27 20:43:00.941347 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:00.941302 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c2f6be6-ed7c-44c1-9203-35245a64cf0d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bslzn\" (UID: \"4c2f6be6-ed7c-44c1-9203-35245a64cf0d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bslzn" Jul 27 20:43:00.941552 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:00.941387 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgc8p\" (UniqueName: \"kubernetes.io/projected/4c2f6be6-ed7c-44c1-9203-35245a64cf0d-kube-api-access-sgc8p\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bslzn\" (UID: \"4c2f6be6-ed7c-44c1-9203-35245a64cf0d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bslzn" Jul 27 20:43:00.941806 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:00.941784 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c2f6be6-ed7c-44c1-9203-35245a64cf0d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bslzn\" (UID: \"4c2f6be6-ed7c-44c1-9203-35245a64cf0d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bslzn" Jul 27 20:43:00.952294 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:00.952271 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgc8p\" (UniqueName: \"kubernetes.io/projected/4c2f6be6-ed7c-44c1-9203-35245a64cf0d-kube-api-access-sgc8p\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bslzn\" (UID: \"4c2f6be6-ed7c-44c1-9203-35245a64cf0d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bslzn" Jul 27 20:43:01.120159 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:01.120067 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bslzn" Jul 27 20:43:01.269251 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:01.269219 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bslzn"] Jul 27 20:43:01.271815 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:43:01.271789 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c2f6be6_ed7c_44c1_9203_35245a64cf0d.slice/crio-76b27044f7c0edd7793bcddd583633c8e020b13d640a8aa4e38fbef93ba98853 WatchSource:0}: Error finding container 76b27044f7c0edd7793bcddd583633c8e020b13d640a8aa4e38fbef93ba98853: Status 404 returned error can't find the container with id 76b27044f7c0edd7793bcddd583633c8e020b13d640a8aa4e38fbef93ba98853 Jul 27 20:43:01.293790 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:01.293760 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bslzn" event={"ID":"4c2f6be6-ed7c-44c1-9203-35245a64cf0d","Type":"ContainerStarted","Data":"76b27044f7c0edd7793bcddd583633c8e020b13d640a8aa4e38fbef93ba98853"} Jul 27 20:43:02.235902 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:02.235871 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5xm4m" Jul 27 20:43:05.254858 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:05.254822 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf8pq" Jul 27 20:43:08.337253 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:08.337215 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bslzn" event={"ID":"4c2f6be6-ed7c-44c1-9203-35245a64cf0d","Type":"ContainerStarted","Data":"933905db0288aec57383e24aad33288b7b7d1f0fd1f0487cdbe25ea9ba7b0cca"} Jul 27 20:43:08.337761 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:08.337413 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bslzn" Jul 27 20:43:08.358385 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:08.358334 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bslzn" podStartSLOduration=2.047816619 podStartE2EDuration="8.358316332s" totalImagesPullingTime="6.310499713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:43:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:43:08.355766152 +0000 UTC m=+518.091746458" watchObservedRunningTime="2026-07-27 20:43:08.358316332 +0000 UTC m=+518.094296618" Jul 27 20:43:09.284072 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:09.284039 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-znhw2" Jul 27 20:43:10.275378 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:10.275338 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5c484c9c79-ttn8p" podUID="4c3952d4-3b16-4463-83e6-a59682647929" containerName="console" containerID="cri-o://cb83d54ee8409d606d12b15913ba24d88c28299dd31593db1332cfb041182f80" gracePeriod=15 Jul 27 20:43:10.531179 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:10.531111 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c484c9c79-ttn8p" Jul 27 20:43:10.635751 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:10.635708 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c3952d4-3b16-4463-83e6-a59682647929-console-oauth-config\") pod \"4c3952d4-3b16-4463-83e6-a59682647929\" (UID: \"4c3952d4-3b16-4463-83e6-a59682647929\") " Jul 27 20:43:10.635958 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:10.635818 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c3952d4-3b16-4463-83e6-a59682647929-trusted-ca-bundle\") pod \"4c3952d4-3b16-4463-83e6-a59682647929\" (UID: \"4c3952d4-3b16-4463-83e6-a59682647929\") " Jul 27 20:43:10.635958 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:10.635856 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c3952d4-3b16-4463-83e6-a59682647929-console-serving-cert\") pod \"4c3952d4-3b16-4463-83e6-a59682647929\" (UID: \"4c3952d4-3b16-4463-83e6-a59682647929\") " Jul 27 20:43:10.635958 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:10.635899 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c3952d4-3b16-4463-83e6-a59682647929-console-config\") pod \"4c3952d4-3b16-4463-83e6-a59682647929\" (UID: \"4c3952d4-3b16-4463-83e6-a59682647929\") " Jul 27 20:43:10.635958 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:10.635932 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cd5rq\" (UniqueName: \"kubernetes.io/projected/4c3952d4-3b16-4463-83e6-a59682647929-kube-api-access-cd5rq\") pod \"4c3952d4-3b16-4463-83e6-a59682647929\" (UID: \"4c3952d4-3b16-4463-83e6-a59682647929\") " Jul 27 20:43:10.636167 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:10.635962 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c3952d4-3b16-4463-83e6-a59682647929-oauth-serving-cert\") pod \"4c3952d4-3b16-4463-83e6-a59682647929\" (UID: \"4c3952d4-3b16-4463-83e6-a59682647929\") " Jul 27 20:43:10.636167 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:10.636007 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c3952d4-3b16-4463-83e6-a59682647929-service-ca\") pod \"4c3952d4-3b16-4463-83e6-a59682647929\" (UID: \"4c3952d4-3b16-4463-83e6-a59682647929\") " Jul 27 20:43:10.636262 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:10.636239 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c3952d4-3b16-4463-83e6-a59682647929-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4c3952d4-3b16-4463-83e6-a59682647929" (UID: "4c3952d4-3b16-4463-83e6-a59682647929"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:43:10.636338 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:10.636305 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c3952d4-3b16-4463-83e6-a59682647929-console-config" (OuterVolumeSpecName: "console-config") pod "4c3952d4-3b16-4463-83e6-a59682647929" (UID: "4c3952d4-3b16-4463-83e6-a59682647929"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:43:10.636395 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:10.636333 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c3952d4-3b16-4463-83e6-a59682647929-trusted-ca-bundle\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:43:10.636519 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:10.636470 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c3952d4-3b16-4463-83e6-a59682647929-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4c3952d4-3b16-4463-83e6-a59682647929" (UID: "4c3952d4-3b16-4463-83e6-a59682647929"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:43:10.636519 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:10.636498 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c3952d4-3b16-4463-83e6-a59682647929-service-ca" (OuterVolumeSpecName: "service-ca") pod "4c3952d4-3b16-4463-83e6-a59682647929" (UID: "4c3952d4-3b16-4463-83e6-a59682647929"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:43:10.638294 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:10.638263 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c3952d4-3b16-4463-83e6-a59682647929-kube-api-access-cd5rq" (OuterVolumeSpecName: "kube-api-access-cd5rq") pod "4c3952d4-3b16-4463-83e6-a59682647929" (UID: "4c3952d4-3b16-4463-83e6-a59682647929"). InnerVolumeSpecName "kube-api-access-cd5rq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:43:10.638701 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:10.638682 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c3952d4-3b16-4463-83e6-a59682647929-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4c3952d4-3b16-4463-83e6-a59682647929" (UID: "4c3952d4-3b16-4463-83e6-a59682647929"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 20:43:10.638790 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:10.638735 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c3952d4-3b16-4463-83e6-a59682647929-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4c3952d4-3b16-4463-83e6-a59682647929" (UID: "4c3952d4-3b16-4463-83e6-a59682647929"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 20:43:10.737430 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:10.737391 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c3952d4-3b16-4463-83e6-a59682647929-oauth-serving-cert\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:43:10.737430 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:10.737425 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c3952d4-3b16-4463-83e6-a59682647929-service-ca\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:43:10.737430 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:10.737438 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c3952d4-3b16-4463-83e6-a59682647929-console-oauth-config\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:43:10.737728 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:10.737451 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c3952d4-3b16-4463-83e6-a59682647929-console-serving-cert\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:43:10.737728 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:10.737462 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c3952d4-3b16-4463-83e6-a59682647929-console-config\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:43:10.737728 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:10.737474 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd5rq\" (UniqueName: \"kubernetes.io/projected/4c3952d4-3b16-4463-83e6-a59682647929-kube-api-access-cd5rq\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:43:11.355889 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:11.355853 2570 generic.go:356] "Generic (PLEG): container finished" podID="4c3952d4-3b16-4463-83e6-a59682647929" containerID="cb83d54ee8409d606d12b15913ba24d88c28299dd31593db1332cfb041182f80" exitCode=0 Jul 27 20:43:11.355889 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:11.355902 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c484c9c79-ttn8p" event={"ID":"4c3952d4-3b16-4463-83e6-a59682647929","Type":"ContainerDied","Data":"cb83d54ee8409d606d12b15913ba24d88c28299dd31593db1332cfb041182f80"} Jul 27 20:43:11.356401 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:11.355927 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c484c9c79-ttn8p" Jul 27 20:43:11.356401 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:11.355943 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c484c9c79-ttn8p" event={"ID":"4c3952d4-3b16-4463-83e6-a59682647929","Type":"ContainerDied","Data":"7da1e6a8e5f5dc7ec0d9dc6ae87a7a692e50e85af85d7c83c4d868d999fddb28"} Jul 27 20:43:11.356401 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:11.355964 2570 scope.go:117] "RemoveContainer" containerID="cb83d54ee8409d606d12b15913ba24d88c28299dd31593db1332cfb041182f80" Jul 27 20:43:11.365358 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:11.365339 2570 scope.go:117] "RemoveContainer" containerID="cb83d54ee8409d606d12b15913ba24d88c28299dd31593db1332cfb041182f80" Jul 27 20:43:11.365680 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:11.365660 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb83d54ee8409d606d12b15913ba24d88c28299dd31593db1332cfb041182f80\": container with ID starting with cb83d54ee8409d606d12b15913ba24d88c28299dd31593db1332cfb041182f80 not found: ID does not exist" containerID="cb83d54ee8409d606d12b15913ba24d88c28299dd31593db1332cfb041182f80" Jul 27 20:43:11.365742 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:11.365687 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb83d54ee8409d606d12b15913ba24d88c28299dd31593db1332cfb041182f80"} err="failed to get container status \"cb83d54ee8409d606d12b15913ba24d88c28299dd31593db1332cfb041182f80\": rpc error: code = NotFound desc = could not find container \"cb83d54ee8409d606d12b15913ba24d88c28299dd31593db1332cfb041182f80\": container with ID starting with cb83d54ee8409d606d12b15913ba24d88c28299dd31593db1332cfb041182f80 not found: ID does not exist" Jul 27 20:43:11.373844 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:11.373808 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c484c9c79-ttn8p"] Jul 27 20:43:11.380224 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:11.380194 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c484c9c79-ttn8p"] Jul 27 20:43:12.854057 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:12.854019 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c3952d4-3b16-4463-83e6-a59682647929" path="/var/lib/kubelet/pods/4c3952d4-3b16-4463-83e6-a59682647929/volumes" Jul 27 20:43:19.345289 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:19.345258 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bslzn" Jul 27 20:43:20.268415 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:20.268380 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7kngr"] Jul 27 20:43:20.268812 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:20.268799 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c3952d4-3b16-4463-83e6-a59682647929" containerName="console" Jul 27 20:43:20.268870 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:20.268814 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c3952d4-3b16-4463-83e6-a59682647929" containerName="console" Jul 27 20:43:20.268910 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:20.268890 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c3952d4-3b16-4463-83e6-a59682647929" containerName="console" Jul 27 20:43:20.273384 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:20.273364 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7kngr" Jul 27 20:43:20.284523 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:20.284491 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7kngr"] Jul 27 20:43:20.427051 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:20.426965 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/39713c56-b9d9-4c5a-b782-b2c0080cce03-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-7kngr\" (UID: \"39713c56-b9d9-4c5a-b782-b2c0080cce03\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7kngr" Jul 27 20:43:20.427051 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:20.427045 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsbmr\" (UniqueName: \"kubernetes.io/projected/39713c56-b9d9-4c5a-b782-b2c0080cce03-kube-api-access-rsbmr\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-7kngr\" (UID: \"39713c56-b9d9-4c5a-b782-b2c0080cce03\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7kngr" Jul 27 20:43:20.528378 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:20.528273 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/39713c56-b9d9-4c5a-b782-b2c0080cce03-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-7kngr\" (UID: \"39713c56-b9d9-4c5a-b782-b2c0080cce03\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7kngr" Jul 27 20:43:20.528378 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:20.528351 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsbmr\" (UniqueName: \"kubernetes.io/projected/39713c56-b9d9-4c5a-b782-b2c0080cce03-kube-api-access-rsbmr\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-7kngr\" (UID: \"39713c56-b9d9-4c5a-b782-b2c0080cce03\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7kngr" Jul 27 20:43:20.528717 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:20.528697 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/39713c56-b9d9-4c5a-b782-b2c0080cce03-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-7kngr\" (UID: \"39713c56-b9d9-4c5a-b782-b2c0080cce03\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7kngr" Jul 27 20:43:20.544308 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:20.544285 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsbmr\" (UniqueName: \"kubernetes.io/projected/39713c56-b9d9-4c5a-b782-b2c0080cce03-kube-api-access-rsbmr\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-7kngr\" (UID: \"39713c56-b9d9-4c5a-b782-b2c0080cce03\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7kngr" Jul 27 20:43:20.585656 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:20.585595 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7kngr" Jul 27 20:43:20.735542 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:20.735515 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7kngr"] Jul 27 20:43:20.737543 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:43:20.737513 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39713c56_b9d9_4c5a_b782_b2c0080cce03.slice/crio-43e8aae04c14d548c6254106321657e0430090b1a986d18f7c7629e90c358ab9 WatchSource:0}: Error finding container 43e8aae04c14d548c6254106321657e0430090b1a986d18f7c7629e90c358ab9: Status 404 returned error can't find the container with id 43e8aae04c14d548c6254106321657e0430090b1a986d18f7c7629e90c358ab9 Jul 27 20:43:21.072418 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.072323 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7kngr"] Jul 27 20:43:21.085271 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.085225 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7kngr"] Jul 27 20:43:21.096925 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.096884 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bslzn"] Jul 27 20:43:21.097148 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.097102 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bslzn" podUID="4c2f6be6-ed7c-44c1-9203-35245a64cf0d" containerName="manager" containerID="cri-o://933905db0288aec57383e24aad33288b7b7d1f0fd1f0487cdbe25ea9ba7b0cca" gracePeriod=2 Jul 27 20:43:21.100532 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.100501 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g7f45"] Jul 27 20:43:21.101140 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.101119 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39713c56-b9d9-4c5a-b782-b2c0080cce03" containerName="manager" Jul 27 20:43:21.101140 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.101140 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="39713c56-b9d9-4c5a-b782-b2c0080cce03" containerName="manager" Jul 27 20:43:21.101299 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.101259 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="39713c56-b9d9-4c5a-b782-b2c0080cce03" containerName="manager" Jul 27 20:43:21.107020 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.105724 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g7f45" Jul 27 20:43:21.108848 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.108811 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bslzn"] Jul 27 20:43:21.114534 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.114465 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g7f45"] Jul 27 20:43:21.120010 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.119980 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf8pq"] Jul 27 20:43:21.121048 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.120718 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf8pq" podUID="82f831ea-3672-4636-be8a-d50cef83fa45" containerName="manager" containerID="cri-o://5231b340d22c541d0491c40c84ce542d9d4bf7ef6d13473e12e5eb293c526237" gracePeriod=2 Jul 27 20:43:21.123849 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.123821 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4zzkt"] Jul 27 20:43:21.124698 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.124657 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c2f6be6-ed7c-44c1-9203-35245a64cf0d" containerName="manager" Jul 27 20:43:21.124698 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.124682 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c2f6be6-ed7c-44c1-9203-35245a64cf0d" containerName="manager" Jul 27 20:43:21.125477 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.124785 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c2f6be6-ed7c-44c1-9203-35245a64cf0d" containerName="manager" Jul 27 20:43:21.128692 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.128639 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf8pq"] Jul 27 20:43:21.128850 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.128771 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4zzkt" Jul 27 20:43:21.134451 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:21.134406 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-bslzn\" is forbidden: User \"system:node:ip-10-0-137-141.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-141.ec2.internal' and this object" podUID="4c2f6be6-ed7c-44c1-9203-35245a64cf0d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bslzn" Jul 27 20:43:21.136542 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:21.136511 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-bslzn\" is forbidden: User \"system:node:ip-10-0-137-141.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-141.ec2.internal' and this object" podUID="4c2f6be6-ed7c-44c1-9203-35245a64cf0d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bslzn" Jul 27 20:43:21.144056 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.144015 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4zzkt"] Jul 27 20:43:21.149554 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.149522 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8k2jd"] Jul 27 20:43:21.150026 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.150009 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82f831ea-3672-4636-be8a-d50cef83fa45" containerName="manager" Jul 27 20:43:21.150119 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.150029 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="82f831ea-3672-4636-be8a-d50cef83fa45" containerName="manager" Jul 27 20:43:21.150176 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.150125 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="82f831ea-3672-4636-be8a-d50cef83fa45" containerName="manager" Jul 27 20:43:21.154646 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.154595 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8k2jd" Jul 27 20:43:21.166106 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:21.166056 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zf8pq\" is forbidden: User \"system:node:ip-10-0-137-141.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-141.ec2.internal' and this object" podUID="82f831ea-3672-4636-be8a-d50cef83fa45" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf8pq" Jul 27 20:43:21.168547 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:21.168478 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-bslzn\" is forbidden: User \"system:node:ip-10-0-137-141.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-141.ec2.internal' and this object" podUID="4c2f6be6-ed7c-44c1-9203-35245a64cf0d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bslzn" Jul 27 20:43:21.171445 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:21.171408 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zf8pq\" is forbidden: User \"system:node:ip-10-0-137-141.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-141.ec2.internal' and this object" podUID="82f831ea-3672-4636-be8a-d50cef83fa45" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf8pq" Jul 27 20:43:21.172748 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.172723 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8k2jd"] Jul 27 20:43:21.235393 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.235360 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8dd034d-be49-4ea0-98be-f9789d930a3f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-g7f45\" (UID: \"b8dd034d-be49-4ea0-98be-f9789d930a3f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g7f45" Jul 27 20:43:21.235521 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.235427 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkdpx\" (UniqueName: \"kubernetes.io/projected/b8dd034d-be49-4ea0-98be-f9789d930a3f-kube-api-access-vkdpx\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-g7f45\" (UID: \"b8dd034d-be49-4ea0-98be-f9789d930a3f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g7f45" Jul 27 20:43:21.235521 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.235490 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/41957af2-b730-4cd2-9d19-6ecb99bd4776-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4zzkt\" (UID: \"41957af2-b730-4cd2-9d19-6ecb99bd4776\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4zzkt" Jul 27 20:43:21.235754 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.235556 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjnnk\" (UniqueName: \"kubernetes.io/projected/41957af2-b730-4cd2-9d19-6ecb99bd4776-kube-api-access-qjnnk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4zzkt\" (UID: \"41957af2-b730-4cd2-9d19-6ecb99bd4776\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4zzkt" Jul 27 20:43:21.235754 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.235620 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chdw9\" (UniqueName: \"kubernetes.io/projected/500a9e69-82e8-4e6e-bbb8-fc6631c306e6-kube-api-access-chdw9\") pod \"limitador-operator-controller-manager-859d7b78bd-8k2jd\" (UID: \"500a9e69-82e8-4e6e-bbb8-fc6631c306e6\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8k2jd" Jul 27 20:43:21.338211 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.338097 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkdpx\" (UniqueName: \"kubernetes.io/projected/b8dd034d-be49-4ea0-98be-f9789d930a3f-kube-api-access-vkdpx\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-g7f45\" (UID: \"b8dd034d-be49-4ea0-98be-f9789d930a3f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g7f45" Jul 27 20:43:21.338211 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.338186 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/41957af2-b730-4cd2-9d19-6ecb99bd4776-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4zzkt\" (UID: \"41957af2-b730-4cd2-9d19-6ecb99bd4776\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4zzkt" Jul 27 20:43:21.338443 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.338215 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjnnk\" (UniqueName: \"kubernetes.io/projected/41957af2-b730-4cd2-9d19-6ecb99bd4776-kube-api-access-qjnnk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4zzkt\" (UID: \"41957af2-b730-4cd2-9d19-6ecb99bd4776\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4zzkt" Jul 27 20:43:21.338443 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.338258 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chdw9\" (UniqueName: \"kubernetes.io/projected/500a9e69-82e8-4e6e-bbb8-fc6631c306e6-kube-api-access-chdw9\") pod \"limitador-operator-controller-manager-859d7b78bd-8k2jd\" (UID: \"500a9e69-82e8-4e6e-bbb8-fc6631c306e6\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8k2jd" Jul 27 20:43:21.338443 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.338353 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8dd034d-be49-4ea0-98be-f9789d930a3f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-g7f45\" (UID: \"b8dd034d-be49-4ea0-98be-f9789d930a3f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g7f45" Jul 27 20:43:21.338827 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.338804 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8dd034d-be49-4ea0-98be-f9789d930a3f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-g7f45\" (UID: \"b8dd034d-be49-4ea0-98be-f9789d930a3f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g7f45" Jul 27 20:43:21.339519 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.339495 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/41957af2-b730-4cd2-9d19-6ecb99bd4776-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4zzkt\" (UID: \"41957af2-b730-4cd2-9d19-6ecb99bd4776\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4zzkt" Jul 27 20:43:21.353250 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.352513 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkdpx\" (UniqueName: \"kubernetes.io/projected/b8dd034d-be49-4ea0-98be-f9789d930a3f-kube-api-access-vkdpx\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-g7f45\" (UID: \"b8dd034d-be49-4ea0-98be-f9789d930a3f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g7f45" Jul 27 20:43:21.355778 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.355750 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chdw9\" (UniqueName: \"kubernetes.io/projected/500a9e69-82e8-4e6e-bbb8-fc6631c306e6-kube-api-access-chdw9\") pod \"limitador-operator-controller-manager-859d7b78bd-8k2jd\" (UID: \"500a9e69-82e8-4e6e-bbb8-fc6631c306e6\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8k2jd" Jul 27 20:43:21.355914 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.355762 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjnnk\" (UniqueName: \"kubernetes.io/projected/41957af2-b730-4cd2-9d19-6ecb99bd4776-kube-api-access-qjnnk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4zzkt\" (UID: \"41957af2-b730-4cd2-9d19-6ecb99bd4776\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4zzkt" Jul 27 20:43:21.374354 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.374322 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf8pq" Jul 27 20:43:21.376591 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:21.376553 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-bslzn\" is forbidden: User \"system:node:ip-10-0-137-141.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-141.ec2.internal' and this object" podUID="4c2f6be6-ed7c-44c1-9203-35245a64cf0d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bslzn" Jul 27 20:43:21.378679 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:21.378645 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zf8pq\" is forbidden: User \"system:node:ip-10-0-137-141.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-141.ec2.internal' and this object" podUID="82f831ea-3672-4636-be8a-d50cef83fa45" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf8pq" Jul 27 20:43:21.385247 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.385214 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bslzn" Jul 27 20:43:21.387322 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:21.387288 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-bslzn\" is forbidden: User \"system:node:ip-10-0-137-141.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-141.ec2.internal' and this object" podUID="4c2f6be6-ed7c-44c1-9203-35245a64cf0d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bslzn" Jul 27 20:43:21.388982 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:21.388958 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zf8pq\" is forbidden: User \"system:node:ip-10-0-137-141.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-141.ec2.internal' and this object" podUID="82f831ea-3672-4636-be8a-d50cef83fa45" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf8pq" Jul 27 20:43:21.413189 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.413143 2570 generic.go:356] "Generic (PLEG): container finished" podID="82f831ea-3672-4636-be8a-d50cef83fa45" containerID="5231b340d22c541d0491c40c84ce542d9d4bf7ef6d13473e12e5eb293c526237" exitCode=0 Jul 27 20:43:21.413391 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.413207 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf8pq" Jul 27 20:43:21.413391 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.413249 2570 scope.go:117] "RemoveContainer" containerID="5231b340d22c541d0491c40c84ce542d9d4bf7ef6d13473e12e5eb293c526237" Jul 27 20:43:21.415399 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:21.415369 2570 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7kngr" container="manager" Jul 27 20:43:21.416799 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:21.416765 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zf8pq\" is forbidden: User \"system:node:ip-10-0-137-141.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-141.ec2.internal' and this object" podUID="82f831ea-3672-4636-be8a-d50cef83fa45" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf8pq" Jul 27 20:43:21.416955 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.416836 2570 generic.go:356] "Generic (PLEG): container finished" podID="4c2f6be6-ed7c-44c1-9203-35245a64cf0d" containerID="933905db0288aec57383e24aad33288b7b7d1f0fd1f0487cdbe25ea9ba7b0cca" exitCode=0 Jul 27 20:43:21.417024 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.416966 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bslzn" Jul 27 20:43:21.418998 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:21.418973 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-bslzn\" is forbidden: User \"system:node:ip-10-0-137-141.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-141.ec2.internal' and this object" podUID="4c2f6be6-ed7c-44c1-9203-35245a64cf0d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bslzn" Jul 27 20:43:21.421003 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:21.420975 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-7kngr\" is forbidden: User \"system:node:ip-10-0-137-141.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-141.ec2.internal' and this object" podUID="39713c56-b9d9-4c5a-b782-b2c0080cce03" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7kngr" Jul 27 20:43:21.422955 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:21.422927 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-bslzn\" is forbidden: User \"system:node:ip-10-0-137-141.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-141.ec2.internal' and this object" podUID="4c2f6be6-ed7c-44c1-9203-35245a64cf0d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bslzn" Jul 27 20:43:21.423499 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.423438 2570 scope.go:117] "RemoveContainer" containerID="5231b340d22c541d0491c40c84ce542d9d4bf7ef6d13473e12e5eb293c526237" Jul 27 20:43:21.423825 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:21.423800 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5231b340d22c541d0491c40c84ce542d9d4bf7ef6d13473e12e5eb293c526237\": container with ID starting with 5231b340d22c541d0491c40c84ce542d9d4bf7ef6d13473e12e5eb293c526237 not found: ID does not exist" containerID="5231b340d22c541d0491c40c84ce542d9d4bf7ef6d13473e12e5eb293c526237" Jul 27 20:43:21.423916 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.423838 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5231b340d22c541d0491c40c84ce542d9d4bf7ef6d13473e12e5eb293c526237"} err="failed to get container status \"5231b340d22c541d0491c40c84ce542d9d4bf7ef6d13473e12e5eb293c526237\": rpc error: code = NotFound desc = could not find container \"5231b340d22c541d0491c40c84ce542d9d4bf7ef6d13473e12e5eb293c526237\": container with ID starting with 5231b340d22c541d0491c40c84ce542d9d4bf7ef6d13473e12e5eb293c526237 not found: ID does not exist" Jul 27 20:43:21.423916 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.423861 2570 scope.go:117] "RemoveContainer" containerID="933905db0288aec57383e24aad33288b7b7d1f0fd1f0487cdbe25ea9ba7b0cca" Jul 27 20:43:21.424647 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:21.424620 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zf8pq\" is forbidden: User \"system:node:ip-10-0-137-141.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-141.ec2.internal' and this object" podUID="82f831ea-3672-4636-be8a-d50cef83fa45" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf8pq" Jul 27 20:43:21.434290 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.434265 2570 scope.go:117] "RemoveContainer" containerID="933905db0288aec57383e24aad33288b7b7d1f0fd1f0487cdbe25ea9ba7b0cca" Jul 27 20:43:21.434683 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:21.434656 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"933905db0288aec57383e24aad33288b7b7d1f0fd1f0487cdbe25ea9ba7b0cca\": container with ID starting with 933905db0288aec57383e24aad33288b7b7d1f0fd1f0487cdbe25ea9ba7b0cca not found: ID does not exist" containerID="933905db0288aec57383e24aad33288b7b7d1f0fd1f0487cdbe25ea9ba7b0cca" Jul 27 20:43:21.434755 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.434688 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"933905db0288aec57383e24aad33288b7b7d1f0fd1f0487cdbe25ea9ba7b0cca"} err="failed to get container status \"933905db0288aec57383e24aad33288b7b7d1f0fd1f0487cdbe25ea9ba7b0cca\": rpc error: code = NotFound desc = could not find container \"933905db0288aec57383e24aad33288b7b7d1f0fd1f0487cdbe25ea9ba7b0cca\": container with ID starting with 933905db0288aec57383e24aad33288b7b7d1f0fd1f0487cdbe25ea9ba7b0cca not found: ID does not exist" Jul 27 20:43:21.439032 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.439007 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sr8vx\" (UniqueName: \"kubernetes.io/projected/82f831ea-3672-4636-be8a-d50cef83fa45-kube-api-access-sr8vx\") pod \"82f831ea-3672-4636-be8a-d50cef83fa45\" (UID: \"82f831ea-3672-4636-be8a-d50cef83fa45\") " Jul 27 20:43:21.441341 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.441313 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f831ea-3672-4636-be8a-d50cef83fa45-kube-api-access-sr8vx" (OuterVolumeSpecName: "kube-api-access-sr8vx") pod "82f831ea-3672-4636-be8a-d50cef83fa45" (UID: "82f831ea-3672-4636-be8a-d50cef83fa45"). InnerVolumeSpecName "kube-api-access-sr8vx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:43:21.522727 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.522685 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g7f45" Jul 27 20:43:21.529649 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.529624 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4zzkt" Jul 27 20:43:21.535869 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.535839 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8k2jd" Jul 27 20:43:21.542011 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.541981 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c2f6be6-ed7c-44c1-9203-35245a64cf0d-extensions-socket-volume\") pod \"4c2f6be6-ed7c-44c1-9203-35245a64cf0d\" (UID: \"4c2f6be6-ed7c-44c1-9203-35245a64cf0d\") " Jul 27 20:43:21.542160 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.542035 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgc8p\" (UniqueName: \"kubernetes.io/projected/4c2f6be6-ed7c-44c1-9203-35245a64cf0d-kube-api-access-sgc8p\") pod \"4c2f6be6-ed7c-44c1-9203-35245a64cf0d\" (UID: \"4c2f6be6-ed7c-44c1-9203-35245a64cf0d\") " Jul 27 20:43:21.542332 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.542314 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr8vx\" (UniqueName: \"kubernetes.io/projected/82f831ea-3672-4636-be8a-d50cef83fa45-kube-api-access-sr8vx\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:43:21.542557 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.542532 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c2f6be6-ed7c-44c1-9203-35245a64cf0d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4c2f6be6-ed7c-44c1-9203-35245a64cf0d" (UID: "4c2f6be6-ed7c-44c1-9203-35245a64cf0d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 20:43:21.544568 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.544541 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c2f6be6-ed7c-44c1-9203-35245a64cf0d-kube-api-access-sgc8p" (OuterVolumeSpecName: "kube-api-access-sgc8p") pod "4c2f6be6-ed7c-44c1-9203-35245a64cf0d" (UID: "4c2f6be6-ed7c-44c1-9203-35245a64cf0d"). InnerVolumeSpecName "kube-api-access-sgc8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:43:21.643371 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.643331 2570 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c2f6be6-ed7c-44c1-9203-35245a64cf0d-extensions-socket-volume\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:43:21.643371 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.643375 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgc8p\" (UniqueName: \"kubernetes.io/projected/4c2f6be6-ed7c-44c1-9203-35245a64cf0d-kube-api-access-sgc8p\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:43:21.691055 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.690951 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g7f45"] Jul 27 20:43:21.694241 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:43:21.694213 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8dd034d_be49_4ea0_98be_f9789d930a3f.slice/crio-6a39d8c8ce106d62d13a04ce8fea2386b8039ddbd6b6f11f6e665bdfce81b9af WatchSource:0}: Error finding container 6a39d8c8ce106d62d13a04ce8fea2386b8039ddbd6b6f11f6e665bdfce81b9af: Status 404 returned error can't find the container with id 6a39d8c8ce106d62d13a04ce8fea2386b8039ddbd6b6f11f6e665bdfce81b9af Jul 27 20:43:21.745771 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:21.745735 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-7kngr\" is forbidden: User \"system:node:ip-10-0-137-141.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-141.ec2.internal' and this object" podUID="39713c56-b9d9-4c5a-b782-b2c0080cce03" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7kngr" Jul 27 20:43:21.747674 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:21.747644 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-bslzn\" is forbidden: User \"system:node:ip-10-0-137-141.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-141.ec2.internal' and this object" podUID="4c2f6be6-ed7c-44c1-9203-35245a64cf0d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bslzn" Jul 27 20:43:21.749534 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:21.749501 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zf8pq\" is forbidden: User \"system:node:ip-10-0-137-141.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-141.ec2.internal' and this object" podUID="82f831ea-3672-4636-be8a-d50cef83fa45" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf8pq" Jul 27 20:43:21.751466 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:21.751442 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-7kngr\" is forbidden: User \"system:node:ip-10-0-137-141.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-141.ec2.internal' and this object" podUID="39713c56-b9d9-4c5a-b782-b2c0080cce03" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7kngr" Jul 27 20:43:21.753371 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:21.753344 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-bslzn\" is forbidden: User \"system:node:ip-10-0-137-141.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-141.ec2.internal' and this object" podUID="4c2f6be6-ed7c-44c1-9203-35245a64cf0d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bslzn" Jul 27 20:43:21.755301 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:21.755267 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zf8pq\" is forbidden: User \"system:node:ip-10-0-137-141.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-141.ec2.internal' and this object" podUID="82f831ea-3672-4636-be8a-d50cef83fa45" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf8pq" Jul 27 20:43:21.925543 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.925513 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8k2jd"] Jul 27 20:43:21.927013 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:43:21.926985 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod500a9e69_82e8_4e6e_bbb8_fc6631c306e6.slice/crio-beb88a804adbd53214e26269b46d110c7c29fc3c7b9afb910a39a2116ecc6c00 WatchSource:0}: Error finding container beb88a804adbd53214e26269b46d110c7c29fc3c7b9afb910a39a2116ecc6c00: Status 404 returned error can't find the container with id beb88a804adbd53214e26269b46d110c7c29fc3c7b9afb910a39a2116ecc6c00 Jul 27 20:43:21.933939 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:21.933913 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4zzkt"] Jul 27 20:43:21.935009 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:43:21.934984 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41957af2_b730_4cd2_9d19_6ecb99bd4776.slice/crio-f4573ea54adea2b69e549e06d3d746e9f311adc745cdd62441cde5ba21cdfebf WatchSource:0}: Error finding container f4573ea54adea2b69e549e06d3d746e9f311adc745cdd62441cde5ba21cdfebf: Status 404 returned error can't find the container with id f4573ea54adea2b69e549e06d3d746e9f311adc745cdd62441cde5ba21cdfebf Jul 27 20:43:22.424986 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:22.424949 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g7f45" event={"ID":"b8dd034d-be49-4ea0-98be-f9789d930a3f","Type":"ContainerStarted","Data":"f56e2718f27016c47fb64813a1eb040a23d67799f9988d37f6af3054a10f4ccc"} Jul 27 20:43:22.425209 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:22.425128 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g7f45" Jul 27 20:43:22.425209 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:22.425180 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g7f45" event={"ID":"b8dd034d-be49-4ea0-98be-f9789d930a3f","Type":"ContainerStarted","Data":"6a39d8c8ce106d62d13a04ce8fea2386b8039ddbd6b6f11f6e665bdfce81b9af"} Jul 27 20:43:22.427374 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:22.427332 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-bslzn\" is forbidden: User \"system:node:ip-10-0-137-141.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-141.ec2.internal' and this object" podUID="4c2f6be6-ed7c-44c1-9203-35245a64cf0d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bslzn" Jul 27 20:43:22.427953 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:22.427923 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7kngr" podUID="39713c56-b9d9-4c5a-b782-b2c0080cce03" containerName="manager" containerID="cri-o://701c28276f178ffd1bb266d09f81430c5dc86fef1095977fdcd313abae6fd1e2" gracePeriod=2 Jul 27 20:43:22.429314 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:22.429279 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zf8pq\" is forbidden: User \"system:node:ip-10-0-137-141.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-141.ec2.internal' and this object" podUID="82f831ea-3672-4636-be8a-d50cef83fa45" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf8pq" Jul 27 20:43:22.429589 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:22.429565 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8k2jd" event={"ID":"500a9e69-82e8-4e6e-bbb8-fc6631c306e6","Type":"ContainerStarted","Data":"d81a7fe000195431f814764a640bae62c132fb69a6f05e5902a692a64d930671"} Jul 27 20:43:22.429708 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:22.429656 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8k2jd" Jul 27 20:43:22.429708 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:22.429682 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8k2jd" event={"ID":"500a9e69-82e8-4e6e-bbb8-fc6631c306e6","Type":"ContainerStarted","Data":"beb88a804adbd53214e26269b46d110c7c29fc3c7b9afb910a39a2116ecc6c00"} Jul 27 20:43:22.431291 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:22.431252 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4zzkt" event={"ID":"41957af2-b730-4cd2-9d19-6ecb99bd4776","Type":"ContainerStarted","Data":"e556b0be0f0c7fda898b47bfc45f5bb3b9abd7b8dcd2e9bd6aa7298ae30695f4"} Jul 27 20:43:22.431412 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:22.431312 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4zzkt" event={"ID":"41957af2-b730-4cd2-9d19-6ecb99bd4776","Type":"ContainerStarted","Data":"f4573ea54adea2b69e549e06d3d746e9f311adc745cdd62441cde5ba21cdfebf"} Jul 27 20:43:22.431412 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:22.431376 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4zzkt" Jul 27 20:43:22.451438 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:22.451385 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g7f45" podStartSLOduration=1.451367443 podStartE2EDuration="1.451367443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:43:22.451031245 +0000 UTC m=+532.187011533" watchObservedRunningTime="2026-07-27 20:43:22.451367443 +0000 UTC m=+532.187347730" Jul 27 20:43:22.452931 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:22.452897 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-7kngr\" is forbidden: User \"system:node:ip-10-0-137-141.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-141.ec2.internal' and this object" podUID="39713c56-b9d9-4c5a-b782-b2c0080cce03" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7kngr" Jul 27 20:43:22.454738 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:22.454711 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-7kngr\" is forbidden: User \"system:node:ip-10-0-137-141.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-141.ec2.internal' and this object" podUID="39713c56-b9d9-4c5a-b782-b2c0080cce03" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7kngr" Jul 27 20:43:22.456442 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:22.456416 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-bslzn\" is forbidden: User \"system:node:ip-10-0-137-141.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-141.ec2.internal' and this object" podUID="4c2f6be6-ed7c-44c1-9203-35245a64cf0d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bslzn" Jul 27 20:43:22.474707 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:22.474674 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zf8pq\" is forbidden: User \"system:node:ip-10-0-137-141.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-141.ec2.internal' and this object" podUID="82f831ea-3672-4636-be8a-d50cef83fa45" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf8pq" Jul 27 20:43:22.474845 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:22.474702 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4zzkt" podStartSLOduration=1.474683258 podStartE2EDuration="1.474683258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:43:22.472717006 +0000 UTC m=+532.208697294" watchObservedRunningTime="2026-07-27 20:43:22.474683258 +0000 UTC m=+532.210663545" Jul 27 20:43:22.502965 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:22.502900 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8k2jd" podStartSLOduration=1.5028759699999998 podStartE2EDuration="1.50287597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:43:22.502192915 +0000 UTC m=+532.238173202" watchObservedRunningTime="2026-07-27 20:43:22.50287597 +0000 UTC m=+532.238856257" Jul 27 20:43:22.691162 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:22.691091 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7kngr" Jul 27 20:43:22.693436 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:22.693387 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-bslzn\" is forbidden: User \"system:node:ip-10-0-137-141.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-141.ec2.internal' and this object" podUID="4c2f6be6-ed7c-44c1-9203-35245a64cf0d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bslzn" Jul 27 20:43:22.695318 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:22.695290 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zf8pq\" is forbidden: User \"system:node:ip-10-0-137-141.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-141.ec2.internal' and this object" podUID="82f831ea-3672-4636-be8a-d50cef83fa45" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf8pq" Jul 27 20:43:22.696937 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:22.696915 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-7kngr\" is forbidden: User \"system:node:ip-10-0-137-141.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-141.ec2.internal' and this object" podUID="39713c56-b9d9-4c5a-b782-b2c0080cce03" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7kngr" Jul 27 20:43:22.854243 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:22.854208 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c2f6be6-ed7c-44c1-9203-35245a64cf0d" path="/var/lib/kubelet/pods/4c2f6be6-ed7c-44c1-9203-35245a64cf0d/volumes" Jul 27 20:43:22.854526 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:22.854513 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82f831ea-3672-4636-be8a-d50cef83fa45" path="/var/lib/kubelet/pods/82f831ea-3672-4636-be8a-d50cef83fa45/volumes" Jul 27 20:43:22.854783 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:22.854764 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rsbmr\" (UniqueName: \"kubernetes.io/projected/39713c56-b9d9-4c5a-b782-b2c0080cce03-kube-api-access-rsbmr\") pod \"39713c56-b9d9-4c5a-b782-b2c0080cce03\" (UID: \"39713c56-b9d9-4c5a-b782-b2c0080cce03\") " Jul 27 20:43:22.854875 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:22.854838 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/39713c56-b9d9-4c5a-b782-b2c0080cce03-extensions-socket-volume\") pod \"39713c56-b9d9-4c5a-b782-b2c0080cce03\" (UID: \"39713c56-b9d9-4c5a-b782-b2c0080cce03\") " Jul 27 20:43:22.855226 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:22.855202 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39713c56-b9d9-4c5a-b782-b2c0080cce03-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "39713c56-b9d9-4c5a-b782-b2c0080cce03" (UID: "39713c56-b9d9-4c5a-b782-b2c0080cce03"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 20:43:22.856992 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:22.856968 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39713c56-b9d9-4c5a-b782-b2c0080cce03-kube-api-access-rsbmr" (OuterVolumeSpecName: "kube-api-access-rsbmr") pod "39713c56-b9d9-4c5a-b782-b2c0080cce03" (UID: "39713c56-b9d9-4c5a-b782-b2c0080cce03"). InnerVolumeSpecName "kube-api-access-rsbmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:43:22.956339 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:22.956242 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsbmr\" (UniqueName: \"kubernetes.io/projected/39713c56-b9d9-4c5a-b782-b2c0080cce03-kube-api-access-rsbmr\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:43:22.956339 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:22.956276 2570 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/39713c56-b9d9-4c5a-b782-b2c0080cce03-extensions-socket-volume\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:43:23.441163 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:23.441127 2570 generic.go:356] "Generic (PLEG): container finished" podID="39713c56-b9d9-4c5a-b782-b2c0080cce03" containerID="701c28276f178ffd1bb266d09f81430c5dc86fef1095977fdcd313abae6fd1e2" exitCode=0 Jul 27 20:43:23.441343 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:23.441197 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7kngr" Jul 27 20:43:23.441343 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:23.441251 2570 scope.go:117] "RemoveContainer" containerID="701c28276f178ffd1bb266d09f81430c5dc86fef1095977fdcd313abae6fd1e2" Jul 27 20:43:23.451123 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:23.451102 2570 scope.go:117] "RemoveContainer" containerID="701c28276f178ffd1bb266d09f81430c5dc86fef1095977fdcd313abae6fd1e2" Jul 27 20:43:23.451454 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:23.451437 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"701c28276f178ffd1bb266d09f81430c5dc86fef1095977fdcd313abae6fd1e2\": container with ID starting with 701c28276f178ffd1bb266d09f81430c5dc86fef1095977fdcd313abae6fd1e2 not found: ID does not exist" containerID="701c28276f178ffd1bb266d09f81430c5dc86fef1095977fdcd313abae6fd1e2" Jul 27 20:43:23.451764 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:23.451476 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"701c28276f178ffd1bb266d09f81430c5dc86fef1095977fdcd313abae6fd1e2"} err="failed to get container status \"701c28276f178ffd1bb266d09f81430c5dc86fef1095977fdcd313abae6fd1e2\": rpc error: code = NotFound desc = could not find container \"701c28276f178ffd1bb266d09f81430c5dc86fef1095977fdcd313abae6fd1e2\": container with ID starting with 701c28276f178ffd1bb266d09f81430c5dc86fef1095977fdcd313abae6fd1e2 not found: ID does not exist" Jul 27 20:43:23.452935 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:23.452908 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-7kngr\" is forbidden: User \"system:node:ip-10-0-137-141.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-141.ec2.internal' and this object" podUID="39713c56-b9d9-4c5a-b782-b2c0080cce03" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7kngr" Jul 27 20:43:24.854440 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:24.854401 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39713c56-b9d9-4c5a-b782-b2c0080cce03" path="/var/lib/kubelet/pods/39713c56-b9d9-4c5a-b782-b2c0080cce03/volumes" Jul 27 20:43:33.443694 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:33.443663 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8k2jd" Jul 27 20:43:33.444180 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:33.443734 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4zzkt" Jul 27 20:43:33.444180 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:33.443878 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g7f45" Jul 27 20:43:33.546975 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:33.546941 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4zzkt"] Jul 27 20:43:33.547186 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:33.547161 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4zzkt" podUID="41957af2-b730-4cd2-9d19-6ecb99bd4776" containerName="manager" containerID="cri-o://e556b0be0f0c7fda898b47bfc45f5bb3b9abd7b8dcd2e9bd6aa7298ae30695f4" gracePeriod=10 Jul 27 20:43:33.809537 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:33.809505 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4zzkt" Jul 27 20:43:33.819123 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:33.819094 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6vzl"] Jul 27 20:43:33.819518 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:33.819505 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41957af2-b730-4cd2-9d19-6ecb99bd4776" containerName="manager" Jul 27 20:43:33.819570 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:33.819521 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="41957af2-b730-4cd2-9d19-6ecb99bd4776" containerName="manager" Jul 27 20:43:33.819630 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:33.819621 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="41957af2-b730-4cd2-9d19-6ecb99bd4776" containerName="manager" Jul 27 20:43:33.822867 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:33.822838 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6vzl" Jul 27 20:43:33.838961 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:33.838931 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6vzl"] Jul 27 20:43:33.949238 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:33.949202 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjnnk\" (UniqueName: \"kubernetes.io/projected/41957af2-b730-4cd2-9d19-6ecb99bd4776-kube-api-access-qjnnk\") pod \"41957af2-b730-4cd2-9d19-6ecb99bd4776\" (UID: \"41957af2-b730-4cd2-9d19-6ecb99bd4776\") " Jul 27 20:43:33.949402 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:33.949338 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/41957af2-b730-4cd2-9d19-6ecb99bd4776-extensions-socket-volume\") pod \"41957af2-b730-4cd2-9d19-6ecb99bd4776\" (UID: \"41957af2-b730-4cd2-9d19-6ecb99bd4776\") " Jul 27 20:43:33.949471 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:33.949456 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvpkn\" (UniqueName: \"kubernetes.io/projected/11104d7e-28dc-48f7-afd9-de980be2d8f1-kube-api-access-zvpkn\") pod \"kuadrant-operator-controller-manager-86878c769f-g6vzl\" (UID: \"11104d7e-28dc-48f7-afd9-de980be2d8f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6vzl" Jul 27 20:43:33.949513 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:33.949479 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/11104d7e-28dc-48f7-afd9-de980be2d8f1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-g6vzl\" (UID: \"11104d7e-28dc-48f7-afd9-de980be2d8f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6vzl" Jul 27 20:43:33.949731 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:33.949698 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41957af2-b730-4cd2-9d19-6ecb99bd4776-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "41957af2-b730-4cd2-9d19-6ecb99bd4776" (UID: "41957af2-b730-4cd2-9d19-6ecb99bd4776"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 20:43:33.951529 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:33.951507 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41957af2-b730-4cd2-9d19-6ecb99bd4776-kube-api-access-qjnnk" (OuterVolumeSpecName: "kube-api-access-qjnnk") pod "41957af2-b730-4cd2-9d19-6ecb99bd4776" (UID: "41957af2-b730-4cd2-9d19-6ecb99bd4776"). InnerVolumeSpecName "kube-api-access-qjnnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:43:34.050560 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:34.050466 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvpkn\" (UniqueName: \"kubernetes.io/projected/11104d7e-28dc-48f7-afd9-de980be2d8f1-kube-api-access-zvpkn\") pod \"kuadrant-operator-controller-manager-86878c769f-g6vzl\" (UID: \"11104d7e-28dc-48f7-afd9-de980be2d8f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6vzl" Jul 27 20:43:34.050560 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:34.050504 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/11104d7e-28dc-48f7-afd9-de980be2d8f1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-g6vzl\" (UID: \"11104d7e-28dc-48f7-afd9-de980be2d8f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6vzl" Jul 27 20:43:34.050560 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:34.050537 2570 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/41957af2-b730-4cd2-9d19-6ecb99bd4776-extensions-socket-volume\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:43:34.050560 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:34.050548 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjnnk\" (UniqueName: \"kubernetes.io/projected/41957af2-b730-4cd2-9d19-6ecb99bd4776-kube-api-access-qjnnk\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:43:34.050934 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:34.050864 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/11104d7e-28dc-48f7-afd9-de980be2d8f1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-g6vzl\" (UID: \"11104d7e-28dc-48f7-afd9-de980be2d8f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6vzl" Jul 27 20:43:34.061626 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:34.061563 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvpkn\" (UniqueName: \"kubernetes.io/projected/11104d7e-28dc-48f7-afd9-de980be2d8f1-kube-api-access-zvpkn\") pod \"kuadrant-operator-controller-manager-86878c769f-g6vzl\" (UID: \"11104d7e-28dc-48f7-afd9-de980be2d8f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6vzl" Jul 27 20:43:34.134836 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:34.134793 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6vzl" Jul 27 20:43:34.284910 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:34.284880 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6vzl"] Jul 27 20:43:34.286718 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:43:34.286691 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11104d7e_28dc_48f7_afd9_de980be2d8f1.slice/crio-aaf5a225a97e7a59660ec1f2b48b81e0d00aec0a4b81202b931e225a867ab4e0 WatchSource:0}: Error finding container aaf5a225a97e7a59660ec1f2b48b81e0d00aec0a4b81202b931e225a867ab4e0: Status 404 returned error can't find the container with id aaf5a225a97e7a59660ec1f2b48b81e0d00aec0a4b81202b931e225a867ab4e0 Jul 27 20:43:34.505793 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:34.505754 2570 generic.go:356] "Generic (PLEG): container finished" podID="41957af2-b730-4cd2-9d19-6ecb99bd4776" containerID="e556b0be0f0c7fda898b47bfc45f5bb3b9abd7b8dcd2e9bd6aa7298ae30695f4" exitCode=0 Jul 27 20:43:34.506230 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:34.505808 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4zzkt" Jul 27 20:43:34.506230 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:34.505853 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4zzkt" event={"ID":"41957af2-b730-4cd2-9d19-6ecb99bd4776","Type":"ContainerDied","Data":"e556b0be0f0c7fda898b47bfc45f5bb3b9abd7b8dcd2e9bd6aa7298ae30695f4"} Jul 27 20:43:34.506230 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:34.505911 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4zzkt" event={"ID":"41957af2-b730-4cd2-9d19-6ecb99bd4776","Type":"ContainerDied","Data":"f4573ea54adea2b69e549e06d3d746e9f311adc745cdd62441cde5ba21cdfebf"} Jul 27 20:43:34.506230 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:34.505945 2570 scope.go:117] "RemoveContainer" containerID="e556b0be0f0c7fda898b47bfc45f5bb3b9abd7b8dcd2e9bd6aa7298ae30695f4" Jul 27 20:43:34.507696 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:34.507665 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6vzl" event={"ID":"11104d7e-28dc-48f7-afd9-de980be2d8f1","Type":"ContainerStarted","Data":"b90fa4a3b30a24ee4a595ee3ec8668db63897df3780b58f33bfbaa518962a979"} Jul 27 20:43:34.507848 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:34.507719 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6vzl" event={"ID":"11104d7e-28dc-48f7-afd9-de980be2d8f1","Type":"ContainerStarted","Data":"aaf5a225a97e7a59660ec1f2b48b81e0d00aec0a4b81202b931e225a867ab4e0"} Jul 27 20:43:34.507848 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:34.507763 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6vzl" Jul 27 20:43:34.515637 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:34.515582 2570 scope.go:117] "RemoveContainer" containerID="e556b0be0f0c7fda898b47bfc45f5bb3b9abd7b8dcd2e9bd6aa7298ae30695f4" Jul 27 20:43:34.515925 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:34.515904 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e556b0be0f0c7fda898b47bfc45f5bb3b9abd7b8dcd2e9bd6aa7298ae30695f4\": container with ID starting with e556b0be0f0c7fda898b47bfc45f5bb3b9abd7b8dcd2e9bd6aa7298ae30695f4 not found: ID does not exist" containerID="e556b0be0f0c7fda898b47bfc45f5bb3b9abd7b8dcd2e9bd6aa7298ae30695f4" Jul 27 20:43:34.515996 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:34.515939 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e556b0be0f0c7fda898b47bfc45f5bb3b9abd7b8dcd2e9bd6aa7298ae30695f4"} err="failed to get container status \"e556b0be0f0c7fda898b47bfc45f5bb3b9abd7b8dcd2e9bd6aa7298ae30695f4\": rpc error: code = NotFound desc = could not find container \"e556b0be0f0c7fda898b47bfc45f5bb3b9abd7b8dcd2e9bd6aa7298ae30695f4\": container with ID starting with e556b0be0f0c7fda898b47bfc45f5bb3b9abd7b8dcd2e9bd6aa7298ae30695f4 not found: ID does not exist" Jul 27 20:43:34.532084 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:34.532034 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6vzl" podStartSLOduration=1.532017532 podStartE2EDuration="1.532017532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:43:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:43:34.52989409 +0000 UTC m=+544.265874384" watchObservedRunningTime="2026-07-27 20:43:34.532017532 +0000 UTC m=+544.267997887" Jul 27 20:43:34.544518 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:34.544484 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4zzkt"] Jul 27 20:43:34.551270 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:34.551243 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4zzkt"] Jul 27 20:43:34.854354 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:34.854323 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41957af2-b730-4cd2-9d19-6ecb99bd4776" path="/var/lib/kubelet/pods/41957af2-b730-4cd2-9d19-6ecb99bd4776/volumes" Jul 27 20:43:45.517186 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:45.517151 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6vzl" Jul 27 20:43:45.572441 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:45.572403 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g7f45"] Jul 27 20:43:45.572922 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:45.572867 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g7f45" podUID="b8dd034d-be49-4ea0-98be-f9789d930a3f" containerName="manager" containerID="cri-o://f56e2718f27016c47fb64813a1eb040a23d67799f9988d37f6af3054a10f4ccc" gracePeriod=10 Jul 27 20:43:45.822654 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:45.822625 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g7f45" Jul 27 20:43:45.859480 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:45.859439 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkdpx\" (UniqueName: \"kubernetes.io/projected/b8dd034d-be49-4ea0-98be-f9789d930a3f-kube-api-access-vkdpx\") pod \"b8dd034d-be49-4ea0-98be-f9789d930a3f\" (UID: \"b8dd034d-be49-4ea0-98be-f9789d930a3f\") " Jul 27 20:43:45.859710 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:45.859506 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8dd034d-be49-4ea0-98be-f9789d930a3f-extensions-socket-volume\") pod \"b8dd034d-be49-4ea0-98be-f9789d930a3f\" (UID: \"b8dd034d-be49-4ea0-98be-f9789d930a3f\") " Jul 27 20:43:45.859885 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:45.859855 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8dd034d-be49-4ea0-98be-f9789d930a3f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b8dd034d-be49-4ea0-98be-f9789d930a3f" (UID: "b8dd034d-be49-4ea0-98be-f9789d930a3f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 20:43:45.861780 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:45.861756 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8dd034d-be49-4ea0-98be-f9789d930a3f-kube-api-access-vkdpx" (OuterVolumeSpecName: "kube-api-access-vkdpx") pod "b8dd034d-be49-4ea0-98be-f9789d930a3f" (UID: "b8dd034d-be49-4ea0-98be-f9789d930a3f"). InnerVolumeSpecName "kube-api-access-vkdpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:43:45.960339 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:45.960296 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkdpx\" (UniqueName: \"kubernetes.io/projected/b8dd034d-be49-4ea0-98be-f9789d930a3f-kube-api-access-vkdpx\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:43:45.960339 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:45.960343 2570 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8dd034d-be49-4ea0-98be-f9789d930a3f-extensions-socket-volume\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:43:46.575055 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:46.575018 2570 generic.go:356] "Generic (PLEG): container finished" podID="b8dd034d-be49-4ea0-98be-f9789d930a3f" containerID="f56e2718f27016c47fb64813a1eb040a23d67799f9988d37f6af3054a10f4ccc" exitCode=0 Jul 27 20:43:46.575539 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:46.575067 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g7f45" Jul 27 20:43:46.575539 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:46.575108 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g7f45" event={"ID":"b8dd034d-be49-4ea0-98be-f9789d930a3f","Type":"ContainerDied","Data":"f56e2718f27016c47fb64813a1eb040a23d67799f9988d37f6af3054a10f4ccc"} Jul 27 20:43:46.575539 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:46.575165 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g7f45" event={"ID":"b8dd034d-be49-4ea0-98be-f9789d930a3f","Type":"ContainerDied","Data":"6a39d8c8ce106d62d13a04ce8fea2386b8039ddbd6b6f11f6e665bdfce81b9af"} Jul 27 20:43:46.575539 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:46.575198 2570 scope.go:117] "RemoveContainer" containerID="f56e2718f27016c47fb64813a1eb040a23d67799f9988d37f6af3054a10f4ccc" Jul 27 20:43:46.585456 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:46.585439 2570 scope.go:117] "RemoveContainer" containerID="f56e2718f27016c47fb64813a1eb040a23d67799f9988d37f6af3054a10f4ccc" Jul 27 20:43:46.585747 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:43:46.585729 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f56e2718f27016c47fb64813a1eb040a23d67799f9988d37f6af3054a10f4ccc\": container with ID starting with f56e2718f27016c47fb64813a1eb040a23d67799f9988d37f6af3054a10f4ccc not found: ID does not exist" containerID="f56e2718f27016c47fb64813a1eb040a23d67799f9988d37f6af3054a10f4ccc" Jul 27 20:43:46.585823 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:46.585756 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f56e2718f27016c47fb64813a1eb040a23d67799f9988d37f6af3054a10f4ccc"} err="failed to get container status \"f56e2718f27016c47fb64813a1eb040a23d67799f9988d37f6af3054a10f4ccc\": rpc error: code = NotFound desc = could not find container \"f56e2718f27016c47fb64813a1eb040a23d67799f9988d37f6af3054a10f4ccc\": container with ID starting with f56e2718f27016c47fb64813a1eb040a23d67799f9988d37f6af3054a10f4ccc not found: ID does not exist" Jul 27 20:43:46.597127 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:46.597105 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g7f45"] Jul 27 20:43:46.603258 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:46.603237 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-g7f45"] Jul 27 20:43:46.853853 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:46.853757 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8dd034d-be49-4ea0-98be-f9789d930a3f" path="/var/lib/kubelet/pods/b8dd034d-be49-4ea0-98be-f9789d930a3f/volumes" Jul 27 20:43:49.868177 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:49.868142 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk"] Jul 27 20:43:49.868663 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:49.868579 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8dd034d-be49-4ea0-98be-f9789d930a3f" containerName="manager" Jul 27 20:43:49.868663 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:49.868614 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8dd034d-be49-4ea0-98be-f9789d930a3f" containerName="manager" Jul 27 20:43:49.868964 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:49.868939 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8dd034d-be49-4ea0-98be-f9789d930a3f" containerName="manager" Jul 27 20:43:49.873693 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:49.873671 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:43:49.876053 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:49.876035 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pl9sp\"" Jul 27 20:43:49.876156 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:49.876038 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 27 20:43:49.882795 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:49.882768 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk"] Jul 27 20:43:49.893622 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:49.893561 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83ed38b7-1ad3-4784-8165-65eb3e4e74a1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-r7skk\" (UID: \"83ed38b7-1ad3-4784-8165-65eb3e4e74a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:43:49.893795 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:49.893636 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83ed38b7-1ad3-4784-8165-65eb3e4e74a1-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-r7skk\" (UID: \"83ed38b7-1ad3-4784-8165-65eb3e4e74a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:43:49.893795 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:49.893735 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83ed38b7-1ad3-4784-8165-65eb3e4e74a1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-r7skk\" (UID: \"83ed38b7-1ad3-4784-8165-65eb3e4e74a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:43:49.893795 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:49.893759 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83ed38b7-1ad3-4784-8165-65eb3e4e74a1-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-r7skk\" (UID: \"83ed38b7-1ad3-4784-8165-65eb3e4e74a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:43:49.893974 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:49.893798 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83ed38b7-1ad3-4784-8165-65eb3e4e74a1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-r7skk\" (UID: \"83ed38b7-1ad3-4784-8165-65eb3e4e74a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:43:49.893974 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:49.893831 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ed38b7-1ad3-4784-8165-65eb3e4e74a1-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-r7skk\" (UID: \"83ed38b7-1ad3-4784-8165-65eb3e4e74a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:43:49.893974 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:49.893860 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hrpk\" (UniqueName: \"kubernetes.io/projected/83ed38b7-1ad3-4784-8165-65eb3e4e74a1-kube-api-access-8hrpk\") pod \"maas-default-gateway-openshift-default-5ccb968c66-r7skk\" (UID: \"83ed38b7-1ad3-4784-8165-65eb3e4e74a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:43:49.894139 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:49.893964 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ed38b7-1ad3-4784-8165-65eb3e4e74a1-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-r7skk\" (UID: \"83ed38b7-1ad3-4784-8165-65eb3e4e74a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:43:49.894139 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:49.894037 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83ed38b7-1ad3-4784-8165-65eb3e4e74a1-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-r7skk\" (UID: \"83ed38b7-1ad3-4784-8165-65eb3e4e74a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:43:49.995291 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:49.995253 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ed38b7-1ad3-4784-8165-65eb3e4e74a1-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-r7skk\" (UID: \"83ed38b7-1ad3-4784-8165-65eb3e4e74a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:43:49.995291 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:49.995300 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hrpk\" (UniqueName: \"kubernetes.io/projected/83ed38b7-1ad3-4784-8165-65eb3e4e74a1-kube-api-access-8hrpk\") pod \"maas-default-gateway-openshift-default-5ccb968c66-r7skk\" (UID: \"83ed38b7-1ad3-4784-8165-65eb3e4e74a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:43:49.995552 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:49.995348 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ed38b7-1ad3-4784-8165-65eb3e4e74a1-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-r7skk\" (UID: \"83ed38b7-1ad3-4784-8165-65eb3e4e74a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:43:49.995552 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:49.995379 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83ed38b7-1ad3-4784-8165-65eb3e4e74a1-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-r7skk\" (UID: \"83ed38b7-1ad3-4784-8165-65eb3e4e74a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:43:49.995552 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:49.995416 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83ed38b7-1ad3-4784-8165-65eb3e4e74a1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-r7skk\" (UID: \"83ed38b7-1ad3-4784-8165-65eb3e4e74a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:43:49.995552 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:49.995453 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83ed38b7-1ad3-4784-8165-65eb3e4e74a1-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-r7skk\" (UID: \"83ed38b7-1ad3-4784-8165-65eb3e4e74a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:43:49.995552 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:49.995513 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83ed38b7-1ad3-4784-8165-65eb3e4e74a1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-r7skk\" (UID: \"83ed38b7-1ad3-4784-8165-65eb3e4e74a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:43:49.995859 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:49.995774 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ed38b7-1ad3-4784-8165-65eb3e4e74a1-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-r7skk\" (UID: \"83ed38b7-1ad3-4784-8165-65eb3e4e74a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:43:49.995859 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:49.995831 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83ed38b7-1ad3-4784-8165-65eb3e4e74a1-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-r7skk\" (UID: \"83ed38b7-1ad3-4784-8165-65eb3e4e74a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:43:49.995969 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:49.995893 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83ed38b7-1ad3-4784-8165-65eb3e4e74a1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-r7skk\" (UID: \"83ed38b7-1ad3-4784-8165-65eb3e4e74a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:43:49.995969 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:49.995927 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83ed38b7-1ad3-4784-8165-65eb3e4e74a1-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-r7skk\" (UID: \"83ed38b7-1ad3-4784-8165-65eb3e4e74a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:43:49.996103 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:49.996065 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ed38b7-1ad3-4784-8165-65eb3e4e74a1-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-r7skk\" (UID: \"83ed38b7-1ad3-4784-8165-65eb3e4e74a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:43:49.996319 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:49.996297 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83ed38b7-1ad3-4784-8165-65eb3e4e74a1-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-r7skk\" (UID: \"83ed38b7-1ad3-4784-8165-65eb3e4e74a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:43:49.996767 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:49.996747 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83ed38b7-1ad3-4784-8165-65eb3e4e74a1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-r7skk\" (UID: \"83ed38b7-1ad3-4784-8165-65eb3e4e74a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:43:49.998383 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:49.998363 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83ed38b7-1ad3-4784-8165-65eb3e4e74a1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-r7skk\" (UID: \"83ed38b7-1ad3-4784-8165-65eb3e4e74a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:43:49.998585 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:49.998565 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83ed38b7-1ad3-4784-8165-65eb3e4e74a1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-r7skk\" (UID: \"83ed38b7-1ad3-4784-8165-65eb3e4e74a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:43:50.003508 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:50.003478 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hrpk\" (UniqueName: \"kubernetes.io/projected/83ed38b7-1ad3-4784-8165-65eb3e4e74a1-kube-api-access-8hrpk\") pod \"maas-default-gateway-openshift-default-5ccb968c66-r7skk\" (UID: \"83ed38b7-1ad3-4784-8165-65eb3e4e74a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:43:50.003641 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:50.003484 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83ed38b7-1ad3-4784-8165-65eb3e4e74a1-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-r7skk\" (UID: \"83ed38b7-1ad3-4784-8165-65eb3e4e74a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:43:50.186987 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:50.186897 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:43:50.342635 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:50.342593 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk"] Jul 27 20:43:50.345186 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:43:50.345143 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ed38b7_1ad3_4784_8165_65eb3e4e74a1.slice/crio-d4dab138a86c76793064b6b93ae37f73005b4a8a68cdc1ad77558694ea2df15f WatchSource:0}: Error finding container d4dab138a86c76793064b6b93ae37f73005b4a8a68cdc1ad77558694ea2df15f: Status 404 returned error can't find the container with id d4dab138a86c76793064b6b93ae37f73005b4a8a68cdc1ad77558694ea2df15f Jul 27 20:43:50.599000 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:50.598967 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" event={"ID":"83ed38b7-1ad3-4784-8165-65eb3e4e74a1","Type":"ContainerStarted","Data":"d4dab138a86c76793064b6b93ae37f73005b4a8a68cdc1ad77558694ea2df15f"} Jul 27 20:43:53.049412 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:53.049366 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 20:43:53.049716 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:53.049440 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 20:43:53.049716 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:53.049490 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 20:43:53.619624 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:53.619563 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" event={"ID":"83ed38b7-1ad3-4784-8165-65eb3e4e74a1","Type":"ContainerStarted","Data":"975d631b523f312cb6ce947c8d05cf781c76d7a7cc3edb8cdbcb54fb8468d18d"} Jul 27 20:43:53.642113 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:53.642057 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" podStartSLOduration=1.9401107629999998 podStartE2EDuration="4.642042119s" totalImagesPullingTime="2.701931356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:43:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:43:53.640781922 +0000 UTC m=+563.376762209" watchObservedRunningTime="2026-07-27 20:43:53.642042119 +0000 UTC m=+563.378022405" Jul 27 20:43:54.187804 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:54.187758 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:43:54.192794 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:54.192762 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:43:54.625469 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:54.625443 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:43:54.626454 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:43:54.626436 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:44:05.827196 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:05.827157 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5kbpv"] Jul 27 20:44:05.831133 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:05.831112 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-5kbpv" Jul 27 20:44:05.833701 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:05.833677 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 27 20:44:05.833701 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:05.833697 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6tzdk\"" Jul 27 20:44:05.838944 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:05.838594 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5kbpv"] Jul 27 20:44:05.923867 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:05.923827 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5kbpv"] Jul 27 20:44:05.950359 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:05.950326 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94fgn\" (UniqueName: \"kubernetes.io/projected/93981c43-5ffa-4a88-bba3-04882b4e75d8-kube-api-access-94fgn\") pod \"limitador-limitador-64c94fd64f-5kbpv\" (UID: \"93981c43-5ffa-4a88-bba3-04882b4e75d8\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5kbpv" Jul 27 20:44:05.950563 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:05.950378 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/93981c43-5ffa-4a88-bba3-04882b4e75d8-config-file\") pod \"limitador-limitador-64c94fd64f-5kbpv\" (UID: \"93981c43-5ffa-4a88-bba3-04882b4e75d8\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5kbpv" Jul 27 20:44:06.051489 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:06.051450 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94fgn\" (UniqueName: \"kubernetes.io/projected/93981c43-5ffa-4a88-bba3-04882b4e75d8-kube-api-access-94fgn\") pod \"limitador-limitador-64c94fd64f-5kbpv\" (UID: \"93981c43-5ffa-4a88-bba3-04882b4e75d8\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5kbpv" Jul 27 20:44:06.051732 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:06.051510 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/93981c43-5ffa-4a88-bba3-04882b4e75d8-config-file\") pod \"limitador-limitador-64c94fd64f-5kbpv\" (UID: \"93981c43-5ffa-4a88-bba3-04882b4e75d8\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5kbpv" Jul 27 20:44:06.052156 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:06.052138 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/93981c43-5ffa-4a88-bba3-04882b4e75d8-config-file\") pod \"limitador-limitador-64c94fd64f-5kbpv\" (UID: \"93981c43-5ffa-4a88-bba3-04882b4e75d8\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5kbpv" Jul 27 20:44:06.059684 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:06.059645 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94fgn\" (UniqueName: \"kubernetes.io/projected/93981c43-5ffa-4a88-bba3-04882b4e75d8-kube-api-access-94fgn\") pod \"limitador-limitador-64c94fd64f-5kbpv\" (UID: \"93981c43-5ffa-4a88-bba3-04882b4e75d8\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5kbpv" Jul 27 20:44:06.142929 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:06.142837 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-5kbpv" Jul 27 20:44:06.292048 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:06.292023 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5kbpv"] Jul 27 20:44:06.293669 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:44:06.293643 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93981c43_5ffa_4a88_bba3_04882b4e75d8.slice/crio-6f7a10c0ad4e3e396736d29d62be49380cbbb3c55d47736a416a9c57bfea440a WatchSource:0}: Error finding container 6f7a10c0ad4e3e396736d29d62be49380cbbb3c55d47736a416a9c57bfea440a: Status 404 returned error can't find the container with id 6f7a10c0ad4e3e396736d29d62be49380cbbb3c55d47736a416a9c57bfea440a Jul 27 20:44:06.696812 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:06.696770 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-5kbpv" event={"ID":"93981c43-5ffa-4a88-bba3-04882b4e75d8","Type":"ContainerStarted","Data":"6f7a10c0ad4e3e396736d29d62be49380cbbb3c55d47736a416a9c57bfea440a"} Jul 27 20:44:06.773583 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:06.773533 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-77dfr"] Jul 27 20:44:06.777681 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:06.777649 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-77dfr" Jul 27 20:44:06.780255 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:06.780220 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5lp2t\"" Jul 27 20:44:06.787895 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:06.787868 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-77dfr"] Jul 27 20:44:06.860121 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:06.859974 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm28x\" (UniqueName: \"kubernetes.io/projected/679e09ca-69f8-4824-a846-79f27a9e3599-kube-api-access-zm28x\") pod \"authorino-75d556bd77-77dfr\" (UID: \"679e09ca-69f8-4824-a846-79f27a9e3599\") " pod="kuadrant-system/authorino-75d556bd77-77dfr" Jul 27 20:44:06.961443 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:06.960848 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm28x\" (UniqueName: \"kubernetes.io/projected/679e09ca-69f8-4824-a846-79f27a9e3599-kube-api-access-zm28x\") pod \"authorino-75d556bd77-77dfr\" (UID: \"679e09ca-69f8-4824-a846-79f27a9e3599\") " pod="kuadrant-system/authorino-75d556bd77-77dfr" Jul 27 20:44:06.975594 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:06.974812 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm28x\" (UniqueName: \"kubernetes.io/projected/679e09ca-69f8-4824-a846-79f27a9e3599-kube-api-access-zm28x\") pod \"authorino-75d556bd77-77dfr\" (UID: \"679e09ca-69f8-4824-a846-79f27a9e3599\") " pod="kuadrant-system/authorino-75d556bd77-77dfr" Jul 27 20:44:07.091725 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:07.091682 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-77dfr" Jul 27 20:44:07.261539 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:07.261476 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-77dfr"] Jul 27 20:44:07.263496 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:44:07.263448 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod679e09ca_69f8_4824_a846_79f27a9e3599.slice/crio-3a23cb2645fb751ce304518d59181d9c7b97eba657104fc8042a0340020f321e WatchSource:0}: Error finding container 3a23cb2645fb751ce304518d59181d9c7b97eba657104fc8042a0340020f321e: Status 404 returned error can't find the container with id 3a23cb2645fb751ce304518d59181d9c7b97eba657104fc8042a0340020f321e Jul 27 20:44:07.705906 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:07.705813 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-77dfr" event={"ID":"679e09ca-69f8-4824-a846-79f27a9e3599","Type":"ContainerStarted","Data":"3a23cb2645fb751ce304518d59181d9c7b97eba657104fc8042a0340020f321e"} Jul 27 20:44:09.721274 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:09.721235 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-5kbpv" event={"ID":"93981c43-5ffa-4a88-bba3-04882b4e75d8","Type":"ContainerStarted","Data":"9b98a52e6d45ab83c9855868cd9e07dc818ef698eac11d18c061b401d17c6cda"} Jul 27 20:44:09.721777 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:09.721304 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-5kbpv" Jul 27 20:44:09.742401 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:09.742331 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-5kbpv" podStartSLOduration=2.034464889 podStartE2EDuration="4.742312435s" totalImagesPullingTime="2.707847546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:44:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:44:09.73894717 +0000 UTC m=+579.474927455" watchObservedRunningTime="2026-07-27 20:44:09.742312435 +0000 UTC m=+579.478292721" Jul 27 20:44:11.735267 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:11.735225 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-77dfr" event={"ID":"679e09ca-69f8-4824-a846-79f27a9e3599","Type":"ContainerStarted","Data":"14a7fd7d48094f1a5ec3c68cdd2e335062644cc16a0985f00b59b00e1f18cac6"} Jul 27 20:44:11.752112 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:11.752056 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-77dfr" podStartSLOduration=2.143368957 podStartE2EDuration="5.752039524s" totalImagesPullingTime="3.608670567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:44:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:44:11.750538147 +0000 UTC m=+581.486518435" watchObservedRunningTime="2026-07-27 20:44:11.752039524 +0000 UTC m=+581.488019809" Jul 27 20:44:20.729267 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:20.729234 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-5kbpv" Jul 27 20:44:22.472437 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:22.472394 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5kbpv"] Jul 27 20:44:22.473004 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:22.472691 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-5kbpv" podUID="93981c43-5ffa-4a88-bba3-04882b4e75d8" containerName="limitador" containerID="cri-o://9b98a52e6d45ab83c9855868cd9e07dc818ef698eac11d18c061b401d17c6cda" gracePeriod=30 Jul 27 20:44:23.024192 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:23.024167 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-5kbpv" Jul 27 20:44:23.113554 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:23.113453 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94fgn\" (UniqueName: \"kubernetes.io/projected/93981c43-5ffa-4a88-bba3-04882b4e75d8-kube-api-access-94fgn\") pod \"93981c43-5ffa-4a88-bba3-04882b4e75d8\" (UID: \"93981c43-5ffa-4a88-bba3-04882b4e75d8\") " Jul 27 20:44:23.113769 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:23.113557 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/93981c43-5ffa-4a88-bba3-04882b4e75d8-config-file\") pod \"93981c43-5ffa-4a88-bba3-04882b4e75d8\" (UID: \"93981c43-5ffa-4a88-bba3-04882b4e75d8\") " Jul 27 20:44:23.113960 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:23.113932 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93981c43-5ffa-4a88-bba3-04882b4e75d8-config-file" (OuterVolumeSpecName: "config-file") pod "93981c43-5ffa-4a88-bba3-04882b4e75d8" (UID: "93981c43-5ffa-4a88-bba3-04882b4e75d8"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:44:23.115926 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:23.115897 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93981c43-5ffa-4a88-bba3-04882b4e75d8-kube-api-access-94fgn" (OuterVolumeSpecName: "kube-api-access-94fgn") pod "93981c43-5ffa-4a88-bba3-04882b4e75d8" (UID: "93981c43-5ffa-4a88-bba3-04882b4e75d8"). InnerVolumeSpecName "kube-api-access-94fgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:44:23.214386 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:23.214348 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94fgn\" (UniqueName: \"kubernetes.io/projected/93981c43-5ffa-4a88-bba3-04882b4e75d8-kube-api-access-94fgn\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:44:23.214386 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:23.214380 2570 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/93981c43-5ffa-4a88-bba3-04882b4e75d8-config-file\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:44:23.804012 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:23.803974 2570 generic.go:356] "Generic (PLEG): container finished" podID="93981c43-5ffa-4a88-bba3-04882b4e75d8" containerID="9b98a52e6d45ab83c9855868cd9e07dc818ef698eac11d18c061b401d17c6cda" exitCode=0 Jul 27 20:44:23.804506 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:23.804044 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-5kbpv" Jul 27 20:44:23.804506 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:23.804057 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-5kbpv" event={"ID":"93981c43-5ffa-4a88-bba3-04882b4e75d8","Type":"ContainerDied","Data":"9b98a52e6d45ab83c9855868cd9e07dc818ef698eac11d18c061b401d17c6cda"} Jul 27 20:44:23.804506 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:23.804142 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-5kbpv" event={"ID":"93981c43-5ffa-4a88-bba3-04882b4e75d8","Type":"ContainerDied","Data":"6f7a10c0ad4e3e396736d29d62be49380cbbb3c55d47736a416a9c57bfea440a"} Jul 27 20:44:23.804506 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:23.804165 2570 scope.go:117] "RemoveContainer" containerID="9b98a52e6d45ab83c9855868cd9e07dc818ef698eac11d18c061b401d17c6cda" Jul 27 20:44:23.814319 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:23.814297 2570 scope.go:117] "RemoveContainer" containerID="9b98a52e6d45ab83c9855868cd9e07dc818ef698eac11d18c061b401d17c6cda" Jul 27 20:44:23.814612 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:44:23.814579 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b98a52e6d45ab83c9855868cd9e07dc818ef698eac11d18c061b401d17c6cda\": container with ID starting with 9b98a52e6d45ab83c9855868cd9e07dc818ef698eac11d18c061b401d17c6cda not found: ID does not exist" containerID="9b98a52e6d45ab83c9855868cd9e07dc818ef698eac11d18c061b401d17c6cda" Jul 27 20:44:23.814690 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:23.814622 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b98a52e6d45ab83c9855868cd9e07dc818ef698eac11d18c061b401d17c6cda"} err="failed to get container status \"9b98a52e6d45ab83c9855868cd9e07dc818ef698eac11d18c061b401d17c6cda\": rpc error: code = NotFound desc = could not find container \"9b98a52e6d45ab83c9855868cd9e07dc818ef698eac11d18c061b401d17c6cda\": container with ID starting with 9b98a52e6d45ab83c9855868cd9e07dc818ef698eac11d18c061b401d17c6cda not found: ID does not exist" Jul 27 20:44:23.830555 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:23.830519 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5kbpv"] Jul 27 20:44:23.834190 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:23.834160 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5kbpv"] Jul 27 20:44:24.854000 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:24.853955 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93981c43-5ffa-4a88-bba3-04882b4e75d8" path="/var/lib/kubelet/pods/93981c43-5ffa-4a88-bba3-04882b4e75d8/volumes" Jul 27 20:44:26.601913 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:26.601872 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-547x8"] Jul 27 20:44:26.602885 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:26.602529 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93981c43-5ffa-4a88-bba3-04882b4e75d8" containerName="limitador" Jul 27 20:44:26.602885 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:26.602550 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="93981c43-5ffa-4a88-bba3-04882b4e75d8" containerName="limitador" Jul 27 20:44:26.602885 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:26.602684 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="93981c43-5ffa-4a88-bba3-04882b4e75d8" containerName="limitador" Jul 27 20:44:26.607786 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:26.607764 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-547x8" Jul 27 20:44:26.610243 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:26.610221 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 27 20:44:26.611136 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:26.611112 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 27 20:44:26.611136 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:26.611123 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 27 20:44:26.611292 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:26.611123 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-x8k28\"" Jul 27 20:44:26.615352 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:26.615331 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-547x8"] Jul 27 20:44:26.646353 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:26.646302 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp784\" (UniqueName: \"kubernetes.io/projected/50f446ce-7287-48c1-a674-ee3bcf97429e-kube-api-access-jp784\") pod \"postgres-69fb4594cf-547x8\" (UID: \"50f446ce-7287-48c1-a674-ee3bcf97429e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-547x8" Jul 27 20:44:26.646548 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:26.646367 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/50f446ce-7287-48c1-a674-ee3bcf97429e-data\") pod \"postgres-69fb4594cf-547x8\" (UID: \"50f446ce-7287-48c1-a674-ee3bcf97429e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-547x8" Jul 27 20:44:26.747467 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:26.747429 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp784\" (UniqueName: \"kubernetes.io/projected/50f446ce-7287-48c1-a674-ee3bcf97429e-kube-api-access-jp784\") pod \"postgres-69fb4594cf-547x8\" (UID: \"50f446ce-7287-48c1-a674-ee3bcf97429e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-547x8" Jul 27 20:44:26.747681 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:26.747475 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/50f446ce-7287-48c1-a674-ee3bcf97429e-data\") pod \"postgres-69fb4594cf-547x8\" (UID: \"50f446ce-7287-48c1-a674-ee3bcf97429e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-547x8" Jul 27 20:44:26.747908 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:26.747889 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/50f446ce-7287-48c1-a674-ee3bcf97429e-data\") pod \"postgres-69fb4594cf-547x8\" (UID: \"50f446ce-7287-48c1-a674-ee3bcf97429e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-547x8" Jul 27 20:44:26.757240 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:26.757196 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp784\" (UniqueName: \"kubernetes.io/projected/50f446ce-7287-48c1-a674-ee3bcf97429e-kube-api-access-jp784\") pod \"postgres-69fb4594cf-547x8\" (UID: \"50f446ce-7287-48c1-a674-ee3bcf97429e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-547x8" Jul 27 20:44:26.919885 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:26.919778 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-547x8" Jul 27 20:44:27.061474 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:27.061442 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-547x8"] Jul 27 20:44:27.062634 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:44:27.062592 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f446ce_7287_48c1_a674_ee3bcf97429e.slice/crio-a3c978547ac92c800de124d99bdda8e98f8d716594eb55c3ccbb20fde1ce9c1d WatchSource:0}: Error finding container a3c978547ac92c800de124d99bdda8e98f8d716594eb55c3ccbb20fde1ce9c1d: Status 404 returned error can't find the container with id a3c978547ac92c800de124d99bdda8e98f8d716594eb55c3ccbb20fde1ce9c1d Jul 27 20:44:27.831811 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:27.831772 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-547x8" event={"ID":"50f446ce-7287-48c1-a674-ee3bcf97429e","Type":"ContainerStarted","Data":"a3c978547ac92c800de124d99bdda8e98f8d716594eb55c3ccbb20fde1ce9c1d"} Jul 27 20:44:31.865188 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:31.865153 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 27 20:44:32.866343 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:32.866292 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-547x8" event={"ID":"50f446ce-7287-48c1-a674-ee3bcf97429e","Type":"ContainerStarted","Data":"9adf09ba99c2da083e8f8e357cb01347118e253236fa1cc2ee4795595d268eac"} Jul 27 20:44:32.866754 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:32.866445 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-547x8" Jul 27 20:44:32.883784 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:32.883723 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-547x8" podStartSLOduration=2.085146473 podStartE2EDuration="6.883705733s" totalImagesPullingTime="4.79855926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:44:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:44:32.881691724 +0000 UTC m=+602.617672011" watchObservedRunningTime="2026-07-27 20:44:32.883705733 +0000 UTC m=+602.619686018" Jul 27 20:44:38.903190 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:38.903159 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-547x8" Jul 27 20:44:39.468781 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:39.468746 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jtnsg"] Jul 27 20:44:39.471651 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:39.471626 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jtnsg" Jul 27 20:44:39.478740 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:39.478707 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jtnsg"] Jul 27 20:44:39.574666 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:39.574629 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gsv2\" (UniqueName: \"kubernetes.io/projected/d580538c-2cfc-4ef8-b5d6-aa290165aaca-kube-api-access-7gsv2\") pod \"authorino-5d4b67c446-jtnsg\" (UID: \"d580538c-2cfc-4ef8-b5d6-aa290165aaca\") " pod="kuadrant-system/authorino-5d4b67c446-jtnsg" Jul 27 20:44:39.648745 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:39.648709 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jtnsg"] Jul 27 20:44:39.649020 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:44:39.649000 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-7gsv2], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-jtnsg" podUID="d580538c-2cfc-4ef8-b5d6-aa290165aaca" Jul 27 20:44:39.675160 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:39.675119 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gsv2\" (UniqueName: \"kubernetes.io/projected/d580538c-2cfc-4ef8-b5d6-aa290165aaca-kube-api-access-7gsv2\") pod \"authorino-5d4b67c446-jtnsg\" (UID: \"d580538c-2cfc-4ef8-b5d6-aa290165aaca\") " pod="kuadrant-system/authorino-5d4b67c446-jtnsg" Jul 27 20:44:39.687981 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:39.687944 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gsv2\" (UniqueName: \"kubernetes.io/projected/d580538c-2cfc-4ef8-b5d6-aa290165aaca-kube-api-access-7gsv2\") pod \"authorino-5d4b67c446-jtnsg\" (UID: \"d580538c-2cfc-4ef8-b5d6-aa290165aaca\") " pod="kuadrant-system/authorino-5d4b67c446-jtnsg" Jul 27 20:44:39.903324 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:39.903237 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jtnsg" Jul 27 20:44:39.909916 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:39.909883 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jtnsg" Jul 27 20:44:39.978266 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:39.978229 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gsv2\" (UniqueName: \"kubernetes.io/projected/d580538c-2cfc-4ef8-b5d6-aa290165aaca-kube-api-access-7gsv2\") pod \"d580538c-2cfc-4ef8-b5d6-aa290165aaca\" (UID: \"d580538c-2cfc-4ef8-b5d6-aa290165aaca\") " Jul 27 20:44:39.980533 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:39.980501 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d580538c-2cfc-4ef8-b5d6-aa290165aaca-kube-api-access-7gsv2" (OuterVolumeSpecName: "kube-api-access-7gsv2") pod "d580538c-2cfc-4ef8-b5d6-aa290165aaca" (UID: "d580538c-2cfc-4ef8-b5d6-aa290165aaca"). InnerVolumeSpecName "kube-api-access-7gsv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:44:40.079334 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:40.079287 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gsv2\" (UniqueName: \"kubernetes.io/projected/d580538c-2cfc-4ef8-b5d6-aa290165aaca-kube-api-access-7gsv2\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:44:40.908525 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:40.908492 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jtnsg" Jul 27 20:44:40.935107 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:40.935055 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jtnsg"] Jul 27 20:44:40.940429 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:40.940399 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jtnsg"] Jul 27 20:44:41.641120 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:41.641085 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-77dfr"] Jul 27 20:44:41.641361 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:41.641333 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-77dfr" podUID="679e09ca-69f8-4824-a846-79f27a9e3599" containerName="authorino" containerID="cri-o://14a7fd7d48094f1a5ec3c68cdd2e335062644cc16a0985f00b59b00e1f18cac6" gracePeriod=30 Jul 27 20:44:41.915983 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:41.915903 2570 generic.go:356] "Generic (PLEG): container finished" podID="679e09ca-69f8-4824-a846-79f27a9e3599" containerID="14a7fd7d48094f1a5ec3c68cdd2e335062644cc16a0985f00b59b00e1f18cac6" exitCode=0 Jul 27 20:44:41.915983 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:41.915963 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-77dfr" event={"ID":"679e09ca-69f8-4824-a846-79f27a9e3599","Type":"ContainerDied","Data":"14a7fd7d48094f1a5ec3c68cdd2e335062644cc16a0985f00b59b00e1f18cac6"} Jul 27 20:44:42.447358 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:42.447322 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-77dfr" Jul 27 20:44:42.601002 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:42.600840 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zm28x\" (UniqueName: \"kubernetes.io/projected/679e09ca-69f8-4824-a846-79f27a9e3599-kube-api-access-zm28x\") pod \"679e09ca-69f8-4824-a846-79f27a9e3599\" (UID: \"679e09ca-69f8-4824-a846-79f27a9e3599\") " Jul 27 20:44:42.603092 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:42.603064 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/679e09ca-69f8-4824-a846-79f27a9e3599-kube-api-access-zm28x" (OuterVolumeSpecName: "kube-api-access-zm28x") pod "679e09ca-69f8-4824-a846-79f27a9e3599" (UID: "679e09ca-69f8-4824-a846-79f27a9e3599"). InnerVolumeSpecName "kube-api-access-zm28x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:44:42.702146 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:42.702093 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm28x\" (UniqueName: \"kubernetes.io/projected/679e09ca-69f8-4824-a846-79f27a9e3599-kube-api-access-zm28x\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:44:42.853939 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:42.853856 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d580538c-2cfc-4ef8-b5d6-aa290165aaca" path="/var/lib/kubelet/pods/d580538c-2cfc-4ef8-b5d6-aa290165aaca/volumes" Jul 27 20:44:42.923026 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:42.922980 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-77dfr" event={"ID":"679e09ca-69f8-4824-a846-79f27a9e3599","Type":"ContainerDied","Data":"3a23cb2645fb751ce304518d59181d9c7b97eba657104fc8042a0340020f321e"} Jul 27 20:44:42.923468 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:42.923045 2570 scope.go:117] "RemoveContainer" containerID="14a7fd7d48094f1a5ec3c68cdd2e335062644cc16a0985f00b59b00e1f18cac6" Jul 27 20:44:42.923468 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:42.923002 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-77dfr" Jul 27 20:44:42.940144 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:42.940116 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-77dfr"] Jul 27 20:44:42.943871 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:42.943844 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-77dfr"] Jul 27 20:44:44.853824 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:44.853788 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="679e09ca-69f8-4824-a846-79f27a9e3599" path="/var/lib/kubelet/pods/679e09ca-69f8-4824-a846-79f27a9e3599/volumes" Jul 27 20:44:46.532900 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:46.532866 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6595df7f5b-7n7v4"] Jul 27 20:44:46.533464 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:46.533438 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="679e09ca-69f8-4824-a846-79f27a9e3599" containerName="authorino" Jul 27 20:44:46.533464 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:46.533464 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="679e09ca-69f8-4824-a846-79f27a9e3599" containerName="authorino" Jul 27 20:44:46.533707 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:46.533569 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="679e09ca-69f8-4824-a846-79f27a9e3599" containerName="authorino" Jul 27 20:44:46.541433 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:46.541407 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6595df7f5b-7n7v4" Jul 27 20:44:46.544248 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:46.544210 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 27 20:44:46.544593 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:46.544527 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 27 20:44:46.544771 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:46.544530 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-chstr\"" Jul 27 20:44:46.548508 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:46.548483 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6595df7f5b-7n7v4"] Jul 27 20:44:46.636516 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:46.636476 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ec30c6d-911e-43f7-b5ae-a09bdd1f6584-webhook-cert\") pod \"maas-controller-6595df7f5b-7n7v4\" (UID: \"3ec30c6d-911e-43f7-b5ae-a09bdd1f6584\") " pod="opendatahub/maas-controller-6595df7f5b-7n7v4" Jul 27 20:44:46.636737 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:46.636546 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clc5j\" (UniqueName: \"kubernetes.io/projected/3ec30c6d-911e-43f7-b5ae-a09bdd1f6584-kube-api-access-clc5j\") pod \"maas-controller-6595df7f5b-7n7v4\" (UID: \"3ec30c6d-911e-43f7-b5ae-a09bdd1f6584\") " pod="opendatahub/maas-controller-6595df7f5b-7n7v4" Jul 27 20:44:46.674755 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:46.674720 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84d5c7b66c-6ffgd"] Jul 27 20:44:46.677692 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:46.677669 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84d5c7b66c-6ffgd" Jul 27 20:44:46.687831 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:46.687808 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84d5c7b66c-6ffgd"] Jul 27 20:44:46.738065 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:46.738023 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ec30c6d-911e-43f7-b5ae-a09bdd1f6584-webhook-cert\") pod \"maas-controller-6595df7f5b-7n7v4\" (UID: \"3ec30c6d-911e-43f7-b5ae-a09bdd1f6584\") " pod="opendatahub/maas-controller-6595df7f5b-7n7v4" Jul 27 20:44:46.738256 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:46.738090 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clc5j\" (UniqueName: \"kubernetes.io/projected/3ec30c6d-911e-43f7-b5ae-a09bdd1f6584-kube-api-access-clc5j\") pod \"maas-controller-6595df7f5b-7n7v4\" (UID: \"3ec30c6d-911e-43f7-b5ae-a09bdd1f6584\") " pod="opendatahub/maas-controller-6595df7f5b-7n7v4" Jul 27 20:44:46.740662 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:46.740641 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ec30c6d-911e-43f7-b5ae-a09bdd1f6584-webhook-cert\") pod \"maas-controller-6595df7f5b-7n7v4\" (UID: \"3ec30c6d-911e-43f7-b5ae-a09bdd1f6584\") " pod="opendatahub/maas-controller-6595df7f5b-7n7v4" Jul 27 20:44:46.746826 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:46.746801 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clc5j\" (UniqueName: \"kubernetes.io/projected/3ec30c6d-911e-43f7-b5ae-a09bdd1f6584-kube-api-access-clc5j\") pod \"maas-controller-6595df7f5b-7n7v4\" (UID: \"3ec30c6d-911e-43f7-b5ae-a09bdd1f6584\") " pod="opendatahub/maas-controller-6595df7f5b-7n7v4" Jul 27 20:44:46.839194 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:46.839088 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd61eab7-045c-4a3a-bdf4-f787b0c002c5-webhook-cert\") pod \"maas-controller-84d5c7b66c-6ffgd\" (UID: \"fd61eab7-045c-4a3a-bdf4-f787b0c002c5\") " pod="opendatahub/maas-controller-84d5c7b66c-6ffgd" Jul 27 20:44:46.839194 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:46.839136 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8qvj\" (UniqueName: \"kubernetes.io/projected/fd61eab7-045c-4a3a-bdf4-f787b0c002c5-kube-api-access-g8qvj\") pod \"maas-controller-84d5c7b66c-6ffgd\" (UID: \"fd61eab7-045c-4a3a-bdf4-f787b0c002c5\") " pod="opendatahub/maas-controller-84d5c7b66c-6ffgd" Jul 27 20:44:46.855132 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:46.855085 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6595df7f5b-7n7v4" Jul 27 20:44:46.940764 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:46.940719 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd61eab7-045c-4a3a-bdf4-f787b0c002c5-webhook-cert\") pod \"maas-controller-84d5c7b66c-6ffgd\" (UID: \"fd61eab7-045c-4a3a-bdf4-f787b0c002c5\") " pod="opendatahub/maas-controller-84d5c7b66c-6ffgd" Jul 27 20:44:46.940929 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:46.940788 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8qvj\" (UniqueName: \"kubernetes.io/projected/fd61eab7-045c-4a3a-bdf4-f787b0c002c5-kube-api-access-g8qvj\") pod \"maas-controller-84d5c7b66c-6ffgd\" (UID: \"fd61eab7-045c-4a3a-bdf4-f787b0c002c5\") " pod="opendatahub/maas-controller-84d5c7b66c-6ffgd" Jul 27 20:44:46.944363 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:46.944089 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd61eab7-045c-4a3a-bdf4-f787b0c002c5-webhook-cert\") pod \"maas-controller-84d5c7b66c-6ffgd\" (UID: \"fd61eab7-045c-4a3a-bdf4-f787b0c002c5\") " pod="opendatahub/maas-controller-84d5c7b66c-6ffgd" Jul 27 20:44:46.950891 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:46.950850 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8qvj\" (UniqueName: \"kubernetes.io/projected/fd61eab7-045c-4a3a-bdf4-f787b0c002c5-kube-api-access-g8qvj\") pod \"maas-controller-84d5c7b66c-6ffgd\" (UID: \"fd61eab7-045c-4a3a-bdf4-f787b0c002c5\") " pod="opendatahub/maas-controller-84d5c7b66c-6ffgd" Jul 27 20:44:46.988542 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:46.988504 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84d5c7b66c-6ffgd" Jul 27 20:44:47.003111 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:47.003084 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6595df7f5b-7n7v4"] Jul 27 20:44:47.004068 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:44:47.004042 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec30c6d_911e_43f7_b5ae_a09bdd1f6584.slice/crio-ef5d28cb2b2a77ca49daa81d4396f3098e88e193846f66cce76ece9b3c5be968 WatchSource:0}: Error finding container ef5d28cb2b2a77ca49daa81d4396f3098e88e193846f66cce76ece9b3c5be968: Status 404 returned error can't find the container with id ef5d28cb2b2a77ca49daa81d4396f3098e88e193846f66cce76ece9b3c5be968 Jul 27 20:44:47.005351 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:47.005331 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 20:44:47.131719 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:47.131694 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84d5c7b66c-6ffgd"] Jul 27 20:44:47.133193 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:44:47.133158 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd61eab7_045c_4a3a_bdf4_f787b0c002c5.slice/crio-86567ace9a7a23813657ed094491c659a7aaa870b48fa74b7fea8612d3e6c939 WatchSource:0}: Error finding container 86567ace9a7a23813657ed094491c659a7aaa870b48fa74b7fea8612d3e6c939: Status 404 returned error can't find the container with id 86567ace9a7a23813657ed094491c659a7aaa870b48fa74b7fea8612d3e6c939 Jul 27 20:44:47.961007 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:47.960630 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84d5c7b66c-6ffgd" event={"ID":"fd61eab7-045c-4a3a-bdf4-f787b0c002c5","Type":"ContainerStarted","Data":"86567ace9a7a23813657ed094491c659a7aaa870b48fa74b7fea8612d3e6c939"} Jul 27 20:44:47.963303 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:47.963234 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6595df7f5b-7n7v4" event={"ID":"3ec30c6d-911e-43f7-b5ae-a09bdd1f6584","Type":"ContainerStarted","Data":"ef5d28cb2b2a77ca49daa81d4396f3098e88e193846f66cce76ece9b3c5be968"} Jul 27 20:44:49.979222 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:49.979183 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84d5c7b66c-6ffgd" event={"ID":"fd61eab7-045c-4a3a-bdf4-f787b0c002c5","Type":"ContainerStarted","Data":"f026b51b00bce9395edea976cdc6e00ad9edefbd2eb2def86ede68b55233656f"} Jul 27 20:44:49.979715 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:49.979256 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84d5c7b66c-6ffgd" Jul 27 20:44:49.980691 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:49.980671 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6595df7f5b-7n7v4" event={"ID":"3ec30c6d-911e-43f7-b5ae-a09bdd1f6584","Type":"ContainerStarted","Data":"9b0646d68f5d9619c758469e905c46c22f27bb2341216d35ad03563d1fa639da"} Jul 27 20:44:49.980789 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:49.980717 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6595df7f5b-7n7v4" Jul 27 20:44:50.000765 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:50.000704 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84d5c7b66c-6ffgd" podStartSLOduration=1.54059168 podStartE2EDuration="4.000683651s" totalImagesPullingTime="2.460091971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:44:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:44:49.998178195 +0000 UTC m=+619.734158482" watchObservedRunningTime="2026-07-27 20:44:50.000683651 +0000 UTC m=+619.736663939" Jul 27 20:44:50.016684 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:44:50.016630 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6595df7f5b-7n7v4" podStartSLOduration=1.416979201 podStartE2EDuration="4.016591697s" totalImagesPullingTime="2.599612496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:44:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:44:50.015695677 +0000 UTC m=+619.751675966" watchObservedRunningTime="2026-07-27 20:44:50.016591697 +0000 UTC m=+619.752571983" Jul 27 20:45:00.988397 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:00.988360 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84d5c7b66c-6ffgd" Jul 27 20:45:00.988949 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:00.988768 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6595df7f5b-7n7v4" Jul 27 20:45:01.050258 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:01.050223 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6595df7f5b-7n7v4"] Jul 27 20:45:01.050476 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:01.050437 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6595df7f5b-7n7v4" podUID="3ec30c6d-911e-43f7-b5ae-a09bdd1f6584" containerName="manager" containerID="cri-o://9b0646d68f5d9619c758469e905c46c22f27bb2341216d35ad03563d1fa639da" gracePeriod=30 Jul 27 20:45:01.304292 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:01.304264 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6595df7f5b-7n7v4" Jul 27 20:45:01.378876 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:01.378836 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clc5j\" (UniqueName: \"kubernetes.io/projected/3ec30c6d-911e-43f7-b5ae-a09bdd1f6584-kube-api-access-clc5j\") pod \"3ec30c6d-911e-43f7-b5ae-a09bdd1f6584\" (UID: \"3ec30c6d-911e-43f7-b5ae-a09bdd1f6584\") " Jul 27 20:45:01.379083 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:01.378889 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ec30c6d-911e-43f7-b5ae-a09bdd1f6584-webhook-cert\") pod \"3ec30c6d-911e-43f7-b5ae-a09bdd1f6584\" (UID: \"3ec30c6d-911e-43f7-b5ae-a09bdd1f6584\") " Jul 27 20:45:01.381305 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:01.381268 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ec30c6d-911e-43f7-b5ae-a09bdd1f6584-kube-api-access-clc5j" (OuterVolumeSpecName: "kube-api-access-clc5j") pod "3ec30c6d-911e-43f7-b5ae-a09bdd1f6584" (UID: "3ec30c6d-911e-43f7-b5ae-a09bdd1f6584"). InnerVolumeSpecName "kube-api-access-clc5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:45:01.381305 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:01.381282 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ec30c6d-911e-43f7-b5ae-a09bdd1f6584-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "3ec30c6d-911e-43f7-b5ae-a09bdd1f6584" (UID: "3ec30c6d-911e-43f7-b5ae-a09bdd1f6584"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 20:45:01.479781 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:01.479742 2570 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ec30c6d-911e-43f7-b5ae-a09bdd1f6584-webhook-cert\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:45:01.479781 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:01.479770 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clc5j\" (UniqueName: \"kubernetes.io/projected/3ec30c6d-911e-43f7-b5ae-a09bdd1f6584-kube-api-access-clc5j\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:45:02.053259 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:02.053222 2570 generic.go:356] "Generic (PLEG): container finished" podID="3ec30c6d-911e-43f7-b5ae-a09bdd1f6584" containerID="9b0646d68f5d9619c758469e905c46c22f27bb2341216d35ad03563d1fa639da" exitCode=0 Jul 27 20:45:02.053724 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:02.053275 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6595df7f5b-7n7v4" Jul 27 20:45:02.053724 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:02.053305 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6595df7f5b-7n7v4" event={"ID":"3ec30c6d-911e-43f7-b5ae-a09bdd1f6584","Type":"ContainerDied","Data":"9b0646d68f5d9619c758469e905c46c22f27bb2341216d35ad03563d1fa639da"} Jul 27 20:45:02.053724 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:02.053361 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6595df7f5b-7n7v4" event={"ID":"3ec30c6d-911e-43f7-b5ae-a09bdd1f6584","Type":"ContainerDied","Data":"ef5d28cb2b2a77ca49daa81d4396f3098e88e193846f66cce76ece9b3c5be968"} Jul 27 20:45:02.053724 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:02.053383 2570 scope.go:117] "RemoveContainer" containerID="9b0646d68f5d9619c758469e905c46c22f27bb2341216d35ad03563d1fa639da" Jul 27 20:45:02.063244 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:02.063223 2570 scope.go:117] "RemoveContainer" containerID="9b0646d68f5d9619c758469e905c46c22f27bb2341216d35ad03563d1fa639da" Jul 27 20:45:02.063528 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:45:02.063508 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b0646d68f5d9619c758469e905c46c22f27bb2341216d35ad03563d1fa639da\": container with ID starting with 9b0646d68f5d9619c758469e905c46c22f27bb2341216d35ad03563d1fa639da not found: ID does not exist" containerID="9b0646d68f5d9619c758469e905c46c22f27bb2341216d35ad03563d1fa639da" Jul 27 20:45:02.063620 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:02.063542 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b0646d68f5d9619c758469e905c46c22f27bb2341216d35ad03563d1fa639da"} err="failed to get container status \"9b0646d68f5d9619c758469e905c46c22f27bb2341216d35ad03563d1fa639da\": rpc error: code = NotFound desc = could not find container \"9b0646d68f5d9619c758469e905c46c22f27bb2341216d35ad03563d1fa639da\": container with ID starting with 9b0646d68f5d9619c758469e905c46c22f27bb2341216d35ad03563d1fa639da not found: ID does not exist" Jul 27 20:45:02.074568 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:02.074537 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6595df7f5b-7n7v4"] Jul 27 20:45:02.080309 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:02.080283 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6595df7f5b-7n7v4"] Jul 27 20:45:02.854232 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:02.854199 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ec30c6d-911e-43f7-b5ae-a09bdd1f6584" path="/var/lib/kubelet/pods/3ec30c6d-911e-43f7-b5ae-a09bdd1f6584/volumes" Jul 27 20:45:22.020821 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:22.020783 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-554975b44f-2rt5h"] Jul 27 20:45:22.021298 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:22.021284 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ec30c6d-911e-43f7-b5ae-a09bdd1f6584" containerName="manager" Jul 27 20:45:22.021360 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:22.021303 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ec30c6d-911e-43f7-b5ae-a09bdd1f6584" containerName="manager" Jul 27 20:45:22.021397 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:22.021367 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ec30c6d-911e-43f7-b5ae-a09bdd1f6584" containerName="manager" Jul 27 20:45:22.032814 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:22.032778 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-554975b44f-2rt5h" Jul 27 20:45:22.034545 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:22.034517 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-554975b44f-2rt5h"] Jul 27 20:45:22.035435 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:22.035216 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 27 20:45:22.170260 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:22.170225 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87qgt\" (UniqueName: \"kubernetes.io/projected/7d0c452f-3cf8-41c4-8018-8c9486409ad9-kube-api-access-87qgt\") pod \"maas-api-554975b44f-2rt5h\" (UID: \"7d0c452f-3cf8-41c4-8018-8c9486409ad9\") " pod="odh-ai-gateway-infra/maas-api-554975b44f-2rt5h" Jul 27 20:45:22.170452 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:22.170275 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d0c452f-3cf8-41c4-8018-8c9486409ad9-maas-api-tls\") pod \"maas-api-554975b44f-2rt5h\" (UID: \"7d0c452f-3cf8-41c4-8018-8c9486409ad9\") " pod="odh-ai-gateway-infra/maas-api-554975b44f-2rt5h" Jul 27 20:45:22.170452 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:22.170335 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d0c452f-3cf8-41c4-8018-8c9486409ad9-openshift-service-ca\") pod \"maas-api-554975b44f-2rt5h\" (UID: \"7d0c452f-3cf8-41c4-8018-8c9486409ad9\") " pod="odh-ai-gateway-infra/maas-api-554975b44f-2rt5h" Jul 27 20:45:22.271017 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:22.270924 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d0c452f-3cf8-41c4-8018-8c9486409ad9-maas-api-tls\") pod \"maas-api-554975b44f-2rt5h\" (UID: \"7d0c452f-3cf8-41c4-8018-8c9486409ad9\") " pod="odh-ai-gateway-infra/maas-api-554975b44f-2rt5h" Jul 27 20:45:22.271017 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:22.270968 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d0c452f-3cf8-41c4-8018-8c9486409ad9-openshift-service-ca\") pod \"maas-api-554975b44f-2rt5h\" (UID: \"7d0c452f-3cf8-41c4-8018-8c9486409ad9\") " pod="odh-ai-gateway-infra/maas-api-554975b44f-2rt5h" Jul 27 20:45:22.271218 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:22.271035 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87qgt\" (UniqueName: \"kubernetes.io/projected/7d0c452f-3cf8-41c4-8018-8c9486409ad9-kube-api-access-87qgt\") pod \"maas-api-554975b44f-2rt5h\" (UID: \"7d0c452f-3cf8-41c4-8018-8c9486409ad9\") " pod="odh-ai-gateway-infra/maas-api-554975b44f-2rt5h" Jul 27 20:45:22.271717 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:22.271631 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d0c452f-3cf8-41c4-8018-8c9486409ad9-openshift-service-ca\") pod \"maas-api-554975b44f-2rt5h\" (UID: \"7d0c452f-3cf8-41c4-8018-8c9486409ad9\") " pod="odh-ai-gateway-infra/maas-api-554975b44f-2rt5h" Jul 27 20:45:22.273573 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:22.273546 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d0c452f-3cf8-41c4-8018-8c9486409ad9-maas-api-tls\") pod \"maas-api-554975b44f-2rt5h\" (UID: \"7d0c452f-3cf8-41c4-8018-8c9486409ad9\") " pod="odh-ai-gateway-infra/maas-api-554975b44f-2rt5h" Jul 27 20:45:22.284447 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:22.284390 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87qgt\" (UniqueName: \"kubernetes.io/projected/7d0c452f-3cf8-41c4-8018-8c9486409ad9-kube-api-access-87qgt\") pod \"maas-api-554975b44f-2rt5h\" (UID: \"7d0c452f-3cf8-41c4-8018-8c9486409ad9\") " pod="odh-ai-gateway-infra/maas-api-554975b44f-2rt5h" Jul 27 20:45:22.347522 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:22.347479 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-554975b44f-2rt5h" Jul 27 20:45:22.526409 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:22.526373 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-554975b44f-2rt5h"] Jul 27 20:45:22.627445 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:22.627412 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-87qg9"] Jul 27 20:45:22.632050 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:22.632032 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-87qg9" Jul 27 20:45:22.634494 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:22.634469 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 27 20:45:22.634494 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:22.634488 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-jlb4m\"" Jul 27 20:45:22.641927 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:22.641903 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-87qg9"] Jul 27 20:45:22.777849 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:22.777741 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceb7cfd9-2198-4f37-8e01-d5f1782c20f2-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-87qg9\" (UID: \"ceb7cfd9-2198-4f37-8e01-d5f1782c20f2\") " pod="openshift-ingress/payload-processing-5cc6cd444b-87qg9" Jul 27 20:45:22.777849 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:22.777792 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssxhp\" (UniqueName: \"kubernetes.io/projected/ceb7cfd9-2198-4f37-8e01-d5f1782c20f2-kube-api-access-ssxhp\") pod \"payload-processing-5cc6cd444b-87qg9\" (UID: \"ceb7cfd9-2198-4f37-8e01-d5f1782c20f2\") " pod="openshift-ingress/payload-processing-5cc6cd444b-87qg9" Jul 27 20:45:22.878533 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:22.878492 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceb7cfd9-2198-4f37-8e01-d5f1782c20f2-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-87qg9\" (UID: \"ceb7cfd9-2198-4f37-8e01-d5f1782c20f2\") " pod="openshift-ingress/payload-processing-5cc6cd444b-87qg9" Jul 27 20:45:22.878533 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:22.878536 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssxhp\" (UniqueName: \"kubernetes.io/projected/ceb7cfd9-2198-4f37-8e01-d5f1782c20f2-kube-api-access-ssxhp\") pod \"payload-processing-5cc6cd444b-87qg9\" (UID: \"ceb7cfd9-2198-4f37-8e01-d5f1782c20f2\") " pod="openshift-ingress/payload-processing-5cc6cd444b-87qg9" Jul 27 20:45:22.879402 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:22.879374 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceb7cfd9-2198-4f37-8e01-d5f1782c20f2-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-87qg9\" (UID: \"ceb7cfd9-2198-4f37-8e01-d5f1782c20f2\") " pod="openshift-ingress/payload-processing-5cc6cd444b-87qg9" Jul 27 20:45:22.887336 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:22.887294 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssxhp\" (UniqueName: \"kubernetes.io/projected/ceb7cfd9-2198-4f37-8e01-d5f1782c20f2-kube-api-access-ssxhp\") pod \"payload-processing-5cc6cd444b-87qg9\" (UID: \"ceb7cfd9-2198-4f37-8e01-d5f1782c20f2\") " pod="openshift-ingress/payload-processing-5cc6cd444b-87qg9" Jul 27 20:45:22.945127 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:22.945094 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-87qg9" Jul 27 20:45:23.131648 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:23.131618 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-87qg9"] Jul 27 20:45:23.133822 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:45:23.133797 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceb7cfd9_2198_4f37_8e01_d5f1782c20f2.slice/crio-aa60036e039c726778a7c5791b565ca3b4d0b02eb4836d13aaf16bd253e0d3c9 WatchSource:0}: Error finding container aa60036e039c726778a7c5791b565ca3b4d0b02eb4836d13aaf16bd253e0d3c9: Status 404 returned error can't find the container with id aa60036e039c726778a7c5791b565ca3b4d0b02eb4836d13aaf16bd253e0d3c9 Jul 27 20:45:23.174878 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:23.174772 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-87qg9" event={"ID":"ceb7cfd9-2198-4f37-8e01-d5f1782c20f2","Type":"ContainerStarted","Data":"aa60036e039c726778a7c5791b565ca3b4d0b02eb4836d13aaf16bd253e0d3c9"} Jul 27 20:45:23.176698 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:23.176585 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-554975b44f-2rt5h" event={"ID":"7d0c452f-3cf8-41c4-8018-8c9486409ad9","Type":"ContainerStarted","Data":"07dda077d4ef8f254d2c2fcfa502060a88cf2f976b95a8ed53eeea314f4e102a"} Jul 27 20:45:25.205326 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:25.205283 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-554975b44f-2rt5h" event={"ID":"7d0c452f-3cf8-41c4-8018-8c9486409ad9","Type":"ContainerStarted","Data":"6da5303c46ac9ef8dfcdfb0de1138d1116a916a282afa93712fa0248b7572fe8"} Jul 27 20:45:25.205744 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:25.205355 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-554975b44f-2rt5h" Jul 27 20:45:25.223563 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:25.223508 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-554975b44f-2rt5h" podStartSLOduration=2.045856211 podStartE2EDuration="4.223490921s" totalImagesPullingTime="2.17763471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:45:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:45:25.221663543 +0000 UTC m=+654.957643832" watchObservedRunningTime="2026-07-27 20:45:25.223490921 +0000 UTC m=+654.959471208" Jul 27 20:45:26.214287 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:26.214248 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-87qg9" event={"ID":"ceb7cfd9-2198-4f37-8e01-d5f1782c20f2","Type":"ContainerStarted","Data":"5da8edb9646a3aec44430b3c800f34cc428ab9c0b9114cbcc933fefdfe135db1"} Jul 27 20:45:26.214802 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:26.214320 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-87qg9" Jul 27 20:45:26.232164 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:26.232106 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-87qg9" podStartSLOduration=2.09930679 podStartE2EDuration="4.232089966s" totalImagesPullingTime="2.132783176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:45:26.228706739 +0000 UTC m=+655.964687027" watchObservedRunningTime="2026-07-27 20:45:26.232089966 +0000 UTC m=+655.968070253" Jul 27 20:45:30.240569 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:30.240533 2570 generic.go:356] "Generic (PLEG): container finished" podID="83ed38b7-1ad3-4784-8165-65eb3e4e74a1" containerID="975d631b523f312cb6ce947c8d05cf781c76d7a7cc3edb8cdbcb54fb8468d18d" exitCode=0 Jul 27 20:45:30.240987 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:30.240626 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" event={"ID":"83ed38b7-1ad3-4784-8165-65eb3e4e74a1","Type":"ContainerDied","Data":"975d631b523f312cb6ce947c8d05cf781c76d7a7cc3edb8cdbcb54fb8468d18d"} Jul 27 20:45:30.241076 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:30.241062 2570 scope.go:117] "RemoveContainer" containerID="975d631b523f312cb6ce947c8d05cf781c76d7a7cc3edb8cdbcb54fb8468d18d" Jul 27 20:45:30.242110 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:30.242080 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 20:45:30.242179 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:30.242151 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 20:45:30.242221 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:30.242184 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 27 20:45:31.219777 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:31.219749 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-554975b44f-2rt5h" Jul 27 20:45:31.250067 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:31.250027 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" event={"ID":"83ed38b7-1ad3-4784-8165-65eb3e4e74a1","Type":"ContainerStarted","Data":"a255332eb61021ee033a24c778587dbf3370a01b514aa228e35352949f7aed92"} Jul 27 20:45:32.187635 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:32.187508 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:45:32.187864 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:32.187664 2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-r7skk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 27 20:45:32.187864 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:32.187710 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" podUID="83ed38b7-1ad3-4784-8165-65eb3e4e74a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 27 20:45:33.187942 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:33.187902 2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-r7skk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 27 20:45:33.188568 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:33.188535 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" podUID="83ed38b7-1ad3-4784-8165-65eb3e4e74a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 27 20:45:34.188059 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:34.188014 2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-r7skk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 27 20:45:34.188720 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:34.188684 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" podUID="83ed38b7-1ad3-4784-8165-65eb3e4e74a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 27 20:45:35.187383 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:35.187348 2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-r7skk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 27 20:45:35.187583 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:35.187443 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" podUID="83ed38b7-1ad3-4784-8165-65eb3e4e74a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 27 20:45:36.191456 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:36.191423 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:45:36.191875 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:36.191732 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:45:36.192537 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:36.192518 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-r7skk" Jul 27 20:45:37.222819 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:37.222777 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-87qg9" Jul 27 20:45:42.763614 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:42.763566 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v"] Jul 27 20:45:42.767222 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:42.767194 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v" Jul 27 20:45:42.770231 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:42.770203 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vxgfz\"" Jul 27 20:45:42.770475 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:42.770457 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 27 20:45:42.771027 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:42.771003 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 27 20:45:42.771646 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:42.771588 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 27 20:45:42.791488 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:42.791456 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v"] Jul 27 20:45:42.802466 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:42.802422 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05e8790d-dbde-4fe7-a9c8-95b7007f1467-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v\" (UID: \"05e8790d-dbde-4fe7-a9c8-95b7007f1467\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v" Jul 27 20:45:42.802466 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:42.802463 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05e8790d-dbde-4fe7-a9c8-95b7007f1467-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v\" (UID: \"05e8790d-dbde-4fe7-a9c8-95b7007f1467\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v" Jul 27 20:45:42.802780 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:42.802483 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b5rf\" (UniqueName: \"kubernetes.io/projected/05e8790d-dbde-4fe7-a9c8-95b7007f1467-kube-api-access-6b5rf\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v\" (UID: \"05e8790d-dbde-4fe7-a9c8-95b7007f1467\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v" Jul 27 20:45:42.802780 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:42.802553 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05e8790d-dbde-4fe7-a9c8-95b7007f1467-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v\" (UID: \"05e8790d-dbde-4fe7-a9c8-95b7007f1467\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v" Jul 27 20:45:42.802780 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:42.802656 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05e8790d-dbde-4fe7-a9c8-95b7007f1467-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v\" (UID: \"05e8790d-dbde-4fe7-a9c8-95b7007f1467\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v" Jul 27 20:45:42.802780 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:42.802708 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05e8790d-dbde-4fe7-a9c8-95b7007f1467-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v\" (UID: \"05e8790d-dbde-4fe7-a9c8-95b7007f1467\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v" Jul 27 20:45:42.903224 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:42.903188 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05e8790d-dbde-4fe7-a9c8-95b7007f1467-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v\" (UID: \"05e8790d-dbde-4fe7-a9c8-95b7007f1467\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v" Jul 27 20:45:42.903437 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:42.903257 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05e8790d-dbde-4fe7-a9c8-95b7007f1467-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v\" (UID: \"05e8790d-dbde-4fe7-a9c8-95b7007f1467\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v" Jul 27 20:45:42.903437 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:42.903290 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05e8790d-dbde-4fe7-a9c8-95b7007f1467-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v\" (UID: \"05e8790d-dbde-4fe7-a9c8-95b7007f1467\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v" Jul 27 20:45:42.903437 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:42.903322 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05e8790d-dbde-4fe7-a9c8-95b7007f1467-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v\" (UID: \"05e8790d-dbde-4fe7-a9c8-95b7007f1467\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v" Jul 27 20:45:42.903437 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:42.903346 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b5rf\" (UniqueName: \"kubernetes.io/projected/05e8790d-dbde-4fe7-a9c8-95b7007f1467-kube-api-access-6b5rf\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v\" (UID: \"05e8790d-dbde-4fe7-a9c8-95b7007f1467\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v" Jul 27 20:45:42.903437 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:42.903410 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05e8790d-dbde-4fe7-a9c8-95b7007f1467-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v\" (UID: \"05e8790d-dbde-4fe7-a9c8-95b7007f1467\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v" Jul 27 20:45:42.903956 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:42.903926 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05e8790d-dbde-4fe7-a9c8-95b7007f1467-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v\" (UID: \"05e8790d-dbde-4fe7-a9c8-95b7007f1467\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v" Jul 27 20:45:42.904151 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:42.903978 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05e8790d-dbde-4fe7-a9c8-95b7007f1467-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v\" (UID: \"05e8790d-dbde-4fe7-a9c8-95b7007f1467\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v" Jul 27 20:45:42.904280 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:42.904054 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05e8790d-dbde-4fe7-a9c8-95b7007f1467-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v\" (UID: \"05e8790d-dbde-4fe7-a9c8-95b7007f1467\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v" Jul 27 20:45:42.905999 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:42.905977 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05e8790d-dbde-4fe7-a9c8-95b7007f1467-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v\" (UID: \"05e8790d-dbde-4fe7-a9c8-95b7007f1467\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v" Jul 27 20:45:42.906414 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:42.906398 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05e8790d-dbde-4fe7-a9c8-95b7007f1467-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v\" (UID: \"05e8790d-dbde-4fe7-a9c8-95b7007f1467\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v" Jul 27 20:45:42.911459 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:42.911433 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b5rf\" (UniqueName: \"kubernetes.io/projected/05e8790d-dbde-4fe7-a9c8-95b7007f1467-kube-api-access-6b5rf\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v\" (UID: \"05e8790d-dbde-4fe7-a9c8-95b7007f1467\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v" Jul 27 20:45:43.079226 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:43.079128 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v" Jul 27 20:45:43.228123 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:43.228096 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v"] Jul 27 20:45:43.229468 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:45:43.229440 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05e8790d_dbde_4fe7_a9c8_95b7007f1467.slice/crio-0cc7c34c86a53c6e6589a08061a3068b1a736e6138e212bf07b79a6f6369b9ee WatchSource:0}: Error finding container 0cc7c34c86a53c6e6589a08061a3068b1a736e6138e212bf07b79a6f6369b9ee: Status 404 returned error can't find the container with id 0cc7c34c86a53c6e6589a08061a3068b1a736e6138e212bf07b79a6f6369b9ee Jul 27 20:45:43.332237 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:43.332155 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v" event={"ID":"05e8790d-dbde-4fe7-a9c8-95b7007f1467","Type":"ContainerStarted","Data":"0cc7c34c86a53c6e6589a08061a3068b1a736e6138e212bf07b79a6f6369b9ee"} Jul 27 20:45:45.347419 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:45.347378 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v" event={"ID":"05e8790d-dbde-4fe7-a9c8-95b7007f1467","Type":"ContainerStarted","Data":"468ab71c91059cd166519b0980146ff26bbb64227543f0e121a0ec1e6294effd"} Jul 27 20:45:45.364918 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:45.364863 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v" podStartSLOduration=2.329850519 podStartE2EDuration="3.364845186s" totalImagesPullingTime="1.034994667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:45:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:45:45.364641227 +0000 UTC m=+675.100621525" watchObservedRunningTime="2026-07-27 20:45:45.364845186 +0000 UTC m=+675.100825473" Jul 27 20:45:51.728401 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:51.728360 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-zwt4w"] Jul 27 20:45:51.731591 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:51.731572 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zwt4w" Jul 27 20:45:51.734106 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:51.734081 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 27 20:45:51.740027 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:51.740003 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-zwt4w"] Jul 27 20:45:51.792364 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:51.792328 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zwt4w\" (UID: \"76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zwt4w" Jul 27 20:45:51.792576 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:51.792383 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zwt4w\" (UID: \"76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zwt4w" Jul 27 20:45:51.792576 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:51.792479 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zwt4w\" (UID: \"76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zwt4w" Jul 27 20:45:51.792576 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:51.792542 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zwt4w\" (UID: \"76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zwt4w" Jul 27 20:45:51.792770 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:51.792582 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zwt4w\" (UID: \"76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zwt4w" Jul 27 20:45:51.792770 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:51.792625 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69pn2\" (UniqueName: \"kubernetes.io/projected/76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0-kube-api-access-69pn2\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zwt4w\" (UID: \"76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zwt4w" Jul 27 20:45:51.894030 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:51.893992 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zwt4w\" (UID: \"76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zwt4w" Jul 27 20:45:51.894030 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:51.894033 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69pn2\" (UniqueName: \"kubernetes.io/projected/76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0-kube-api-access-69pn2\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zwt4w\" (UID: \"76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zwt4w" Jul 27 20:45:51.894312 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:51.894092 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zwt4w\" (UID: \"76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zwt4w" Jul 27 20:45:51.894312 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:51.894137 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zwt4w\" (UID: \"76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zwt4w" Jul 27 20:45:51.894312 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:51.894191 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zwt4w\" (UID: \"76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zwt4w" Jul 27 20:45:51.894312 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:51.894240 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zwt4w\" (UID: \"76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zwt4w" Jul 27 20:45:51.894841 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:51.894782 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zwt4w\" (UID: \"76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zwt4w" Jul 27 20:45:51.895454 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:51.895433 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zwt4w\" (UID: \"76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zwt4w" Jul 27 20:45:51.895862 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:51.895838 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zwt4w\" (UID: \"76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zwt4w" Jul 27 20:45:51.897174 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:51.897149 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zwt4w\" (UID: \"76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zwt4w" Jul 27 20:45:51.897355 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:51.897334 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zwt4w\" (UID: \"76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zwt4w" Jul 27 20:45:51.902525 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:51.902491 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69pn2\" (UniqueName: \"kubernetes.io/projected/76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0-kube-api-access-69pn2\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zwt4w\" (UID: \"76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zwt4w" Jul 27 20:45:52.043593 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:52.043482 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zwt4w" Jul 27 20:45:52.194512 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:52.194483 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-zwt4w"] Jul 27 20:45:52.195642 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:45:52.195613 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76e6f21d_36e6_4ee0_ab0c_48669bb6e9d0.slice/crio-4266bd268fabf0a9341c7d70c1c536734873d9b1982c904e4be2d8c6df8b26eb WatchSource:0}: Error finding container 4266bd268fabf0a9341c7d70c1c536734873d9b1982c904e4be2d8c6df8b26eb: Status 404 returned error can't find the container with id 4266bd268fabf0a9341c7d70c1c536734873d9b1982c904e4be2d8c6df8b26eb Jul 27 20:45:52.398053 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:52.398021 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zwt4w" event={"ID":"76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0","Type":"ContainerStarted","Data":"4266bd268fabf0a9341c7d70c1c536734873d9b1982c904e4be2d8c6df8b26eb"} Jul 27 20:45:53.080207 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:53.080163 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v" Jul 27 20:45:53.080576 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:53.080238 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v" Jul 27 20:45:53.092896 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:53.092867 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v" Jul 27 20:45:53.405537 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:53.405439 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zwt4w" event={"ID":"76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0","Type":"ContainerStarted","Data":"f454c7077a6aa87ab04c7e8f159de9b83c4e593015500d2524d79a928e48ce11"} Jul 27 20:45:53.417160 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:53.417124 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v" Jul 27 20:45:53.423858 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:45:53.423805 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zwt4w" podStartSLOduration=2.251046765 podStartE2EDuration="2.423789964s" totalImagesPullingTime="172.743199ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:45:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:45:53.421203974 +0000 UTC m=+683.157184262" watchObservedRunningTime="2026-07-27 20:45:53.423789964 +0000 UTC m=+683.159770251" Jul 27 20:46:02.044116 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:46:02.044071 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zwt4w" Jul 27 20:46:02.044561 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:46:02.044177 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zwt4w" Jul 27 20:46:02.056629 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:46:02.056581 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zwt4w" Jul 27 20:46:02.468243 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:46:02.468213 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zwt4w" Jul 27 20:48:28.447092 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:48:28.447056 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84d5c7b66c-6ffgd"] Jul 27 20:48:28.447678 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:48:28.447317 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-84d5c7b66c-6ffgd" podUID="fd61eab7-045c-4a3a-bdf4-f787b0c002c5" containerName="manager" containerID="cri-o://f026b51b00bce9395edea976cdc6e00ad9edefbd2eb2def86ede68b55233656f" gracePeriod=30 Jul 27 20:48:28.726261 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:48:28.726232 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84d5c7b66c-6ffgd" Jul 27 20:48:28.810559 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:48:28.810524 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd61eab7-045c-4a3a-bdf4-f787b0c002c5-webhook-cert\") pod \"fd61eab7-045c-4a3a-bdf4-f787b0c002c5\" (UID: \"fd61eab7-045c-4a3a-bdf4-f787b0c002c5\") " Jul 27 20:48:28.810784 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:48:28.810687 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8qvj\" (UniqueName: \"kubernetes.io/projected/fd61eab7-045c-4a3a-bdf4-f787b0c002c5-kube-api-access-g8qvj\") pod \"fd61eab7-045c-4a3a-bdf4-f787b0c002c5\" (UID: \"fd61eab7-045c-4a3a-bdf4-f787b0c002c5\") " Jul 27 20:48:28.812945 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:48:28.812908 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd61eab7-045c-4a3a-bdf4-f787b0c002c5-kube-api-access-g8qvj" (OuterVolumeSpecName: "kube-api-access-g8qvj") pod "fd61eab7-045c-4a3a-bdf4-f787b0c002c5" (UID: "fd61eab7-045c-4a3a-bdf4-f787b0c002c5"). InnerVolumeSpecName "kube-api-access-g8qvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:48:28.812945 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:48:28.812907 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd61eab7-045c-4a3a-bdf4-f787b0c002c5-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "fd61eab7-045c-4a3a-bdf4-f787b0c002c5" (UID: "fd61eab7-045c-4a3a-bdf4-f787b0c002c5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 20:48:28.912467 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:48:28.912419 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8qvj\" (UniqueName: \"kubernetes.io/projected/fd61eab7-045c-4a3a-bdf4-f787b0c002c5-kube-api-access-g8qvj\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:48:28.912467 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:48:28.912465 2570 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd61eab7-045c-4a3a-bdf4-f787b0c002c5-webhook-cert\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:48:29.330971 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:48:29.330931 2570 generic.go:356] "Generic (PLEG): container finished" podID="fd61eab7-045c-4a3a-bdf4-f787b0c002c5" containerID="f026b51b00bce9395edea976cdc6e00ad9edefbd2eb2def86ede68b55233656f" exitCode=0 Jul 27 20:48:29.331188 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:48:29.330993 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84d5c7b66c-6ffgd" Jul 27 20:48:29.331188 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:48:29.331015 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84d5c7b66c-6ffgd" event={"ID":"fd61eab7-045c-4a3a-bdf4-f787b0c002c5","Type":"ContainerDied","Data":"f026b51b00bce9395edea976cdc6e00ad9edefbd2eb2def86ede68b55233656f"} Jul 27 20:48:29.331188 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:48:29.331067 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84d5c7b66c-6ffgd" event={"ID":"fd61eab7-045c-4a3a-bdf4-f787b0c002c5","Type":"ContainerDied","Data":"86567ace9a7a23813657ed094491c659a7aaa870b48fa74b7fea8612d3e6c939"} Jul 27 20:48:29.331188 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:48:29.331096 2570 scope.go:117] "RemoveContainer" containerID="f026b51b00bce9395edea976cdc6e00ad9edefbd2eb2def86ede68b55233656f" Jul 27 20:48:29.341037 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:48:29.341015 2570 scope.go:117] "RemoveContainer" containerID="f026b51b00bce9395edea976cdc6e00ad9edefbd2eb2def86ede68b55233656f" Jul 27 20:48:29.341349 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:48:29.341332 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f026b51b00bce9395edea976cdc6e00ad9edefbd2eb2def86ede68b55233656f\": container with ID starting with f026b51b00bce9395edea976cdc6e00ad9edefbd2eb2def86ede68b55233656f not found: ID does not exist" containerID="f026b51b00bce9395edea976cdc6e00ad9edefbd2eb2def86ede68b55233656f" Jul 27 20:48:29.341405 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:48:29.341366 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f026b51b00bce9395edea976cdc6e00ad9edefbd2eb2def86ede68b55233656f"} err="failed to get container status \"f026b51b00bce9395edea976cdc6e00ad9edefbd2eb2def86ede68b55233656f\": rpc error: code = NotFound desc = could not find container \"f026b51b00bce9395edea976cdc6e00ad9edefbd2eb2def86ede68b55233656f\": container with ID starting with f026b51b00bce9395edea976cdc6e00ad9edefbd2eb2def86ede68b55233656f not found: ID does not exist" Jul 27 20:48:29.349673 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:48:29.349638 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84d5c7b66c-6ffgd"] Jul 27 20:48:29.352847 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:48:29.352820 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-84d5c7b66c-6ffgd"] Jul 27 20:48:30.862877 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:48:30.862831 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd61eab7-045c-4a3a-bdf4-f787b0c002c5" path="/var/lib/kubelet/pods/fd61eab7-045c-4a3a-bdf4-f787b0c002c5/volumes" Jul 27 20:49:30.236490 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:30.236383 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84d5c7b66c-z8qvd"] Jul 27 20:49:30.237070 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:30.236888 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd61eab7-045c-4a3a-bdf4-f787b0c002c5" containerName="manager" Jul 27 20:49:30.237070 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:30.236902 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd61eab7-045c-4a3a-bdf4-f787b0c002c5" containerName="manager" Jul 27 20:49:30.237070 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:30.237037 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd61eab7-045c-4a3a-bdf4-f787b0c002c5" containerName="manager" Jul 27 20:49:30.240189 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:30.240158 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84d5c7b66c-z8qvd" Jul 27 20:49:30.243422 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:30.243396 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 27 20:49:30.243564 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:30.243428 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 27 20:49:30.244277 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:30.244260 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-chstr\"" Jul 27 20:49:30.252291 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:30.252266 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84d5c7b66c-z8qvd"] Jul 27 20:49:30.391189 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:30.391154 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z77s\" (UniqueName: \"kubernetes.io/projected/5abd0e8b-5a4c-4117-be05-72d39fc316d6-kube-api-access-7z77s\") pod \"maas-controller-84d5c7b66c-z8qvd\" (UID: \"5abd0e8b-5a4c-4117-be05-72d39fc316d6\") " pod="opendatahub/maas-controller-84d5c7b66c-z8qvd" Jul 27 20:49:30.391373 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:30.391237 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5abd0e8b-5a4c-4117-be05-72d39fc316d6-webhook-cert\") pod \"maas-controller-84d5c7b66c-z8qvd\" (UID: \"5abd0e8b-5a4c-4117-be05-72d39fc316d6\") " pod="opendatahub/maas-controller-84d5c7b66c-z8qvd" Jul 27 20:49:30.492365 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:30.492273 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z77s\" (UniqueName: \"kubernetes.io/projected/5abd0e8b-5a4c-4117-be05-72d39fc316d6-kube-api-access-7z77s\") pod \"maas-controller-84d5c7b66c-z8qvd\" (UID: \"5abd0e8b-5a4c-4117-be05-72d39fc316d6\") " pod="opendatahub/maas-controller-84d5c7b66c-z8qvd" Jul 27 20:49:30.492365 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:30.492342 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5abd0e8b-5a4c-4117-be05-72d39fc316d6-webhook-cert\") pod \"maas-controller-84d5c7b66c-z8qvd\" (UID: \"5abd0e8b-5a4c-4117-be05-72d39fc316d6\") " pod="opendatahub/maas-controller-84d5c7b66c-z8qvd" Jul 27 20:49:30.495065 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:30.495040 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5abd0e8b-5a4c-4117-be05-72d39fc316d6-webhook-cert\") pod \"maas-controller-84d5c7b66c-z8qvd\" (UID: \"5abd0e8b-5a4c-4117-be05-72d39fc316d6\") " pod="opendatahub/maas-controller-84d5c7b66c-z8qvd" Jul 27 20:49:30.504831 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:30.504808 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z77s\" (UniqueName: \"kubernetes.io/projected/5abd0e8b-5a4c-4117-be05-72d39fc316d6-kube-api-access-7z77s\") pod \"maas-controller-84d5c7b66c-z8qvd\" (UID: \"5abd0e8b-5a4c-4117-be05-72d39fc316d6\") " pod="opendatahub/maas-controller-84d5c7b66c-z8qvd" Jul 27 20:49:30.591968 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:30.591931 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84d5c7b66c-z8qvd" Jul 27 20:49:30.746545 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:30.746513 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84d5c7b66c-z8qvd"] Jul 27 20:49:30.748425 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:49:30.748397 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5abd0e8b_5a4c_4117_be05_72d39fc316d6.slice/crio-e3cfbc252ad26f024245818eaf904d8b0c50e6ad5f5a59bfc70f1fb4a91de3fd WatchSource:0}: Error finding container e3cfbc252ad26f024245818eaf904d8b0c50e6ad5f5a59bfc70f1fb4a91de3fd: Status 404 returned error can't find the container with id e3cfbc252ad26f024245818eaf904d8b0c50e6ad5f5a59bfc70f1fb4a91de3fd Jul 27 20:49:31.121418 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:31.121395 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 27 20:49:31.699148 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:31.699114 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84d5c7b66c-z8qvd" event={"ID":"5abd0e8b-5a4c-4117-be05-72d39fc316d6","Type":"ContainerStarted","Data":"76b6379a97c95bff8c4d6e9560aacc28893411b91673a9f4ba5cfccc60a6cf8b"} Jul 27 20:49:31.699583 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:31.699162 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84d5c7b66c-z8qvd" event={"ID":"5abd0e8b-5a4c-4117-be05-72d39fc316d6","Type":"ContainerStarted","Data":"e3cfbc252ad26f024245818eaf904d8b0c50e6ad5f5a59bfc70f1fb4a91de3fd"} Jul 27 20:49:31.699583 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:31.699219 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84d5c7b66c-z8qvd" Jul 27 20:49:31.718322 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:31.718263 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84d5c7b66c-z8qvd" podStartSLOduration=1.349823054 podStartE2EDuration="1.71824929s" totalImagesPullingTime="368.426236ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:49:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:49:31.718157451 +0000 UTC m=+901.454137749" watchObservedRunningTime="2026-07-27 20:49:31.71824929 +0000 UTC m=+901.454229577" Jul 27 20:49:42.707883 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:42.707843 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84d5c7b66c-z8qvd" Jul 27 20:49:47.350916 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:47.350878 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84d5c7b66c-z8qvd"] Jul 27 20:49:47.351509 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:47.351078 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-84d5c7b66c-z8qvd" podUID="5abd0e8b-5a4c-4117-be05-72d39fc316d6" containerName="manager" containerID="cri-o://76b6379a97c95bff8c4d6e9560aacc28893411b91673a9f4ba5cfccc60a6cf8b" gracePeriod=30 Jul 27 20:49:47.610700 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:47.610624 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84d5c7b66c-z8qvd" Jul 27 20:49:47.658713 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:47.658680 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5abd0e8b-5a4c-4117-be05-72d39fc316d6-webhook-cert\") pod \"5abd0e8b-5a4c-4117-be05-72d39fc316d6\" (UID: \"5abd0e8b-5a4c-4117-be05-72d39fc316d6\") " Jul 27 20:49:47.658907 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:47.658730 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7z77s\" (UniqueName: \"kubernetes.io/projected/5abd0e8b-5a4c-4117-be05-72d39fc316d6-kube-api-access-7z77s\") pod \"5abd0e8b-5a4c-4117-be05-72d39fc316d6\" (UID: \"5abd0e8b-5a4c-4117-be05-72d39fc316d6\") " Jul 27 20:49:47.660937 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:47.660908 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5abd0e8b-5a4c-4117-be05-72d39fc316d6-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "5abd0e8b-5a4c-4117-be05-72d39fc316d6" (UID: "5abd0e8b-5a4c-4117-be05-72d39fc316d6"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 20:49:47.661068 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:47.660964 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5abd0e8b-5a4c-4117-be05-72d39fc316d6-kube-api-access-7z77s" (OuterVolumeSpecName: "kube-api-access-7z77s") pod "5abd0e8b-5a4c-4117-be05-72d39fc316d6" (UID: "5abd0e8b-5a4c-4117-be05-72d39fc316d6"). InnerVolumeSpecName "kube-api-access-7z77s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:49:47.759713 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:47.759660 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z77s\" (UniqueName: \"kubernetes.io/projected/5abd0e8b-5a4c-4117-be05-72d39fc316d6-kube-api-access-7z77s\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:49:47.759713 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:47.759708 2570 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5abd0e8b-5a4c-4117-be05-72d39fc316d6-webhook-cert\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:49:47.795438 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:47.795407 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84d5c7b66c-z8qvd" Jul 27 20:49:47.795438 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:47.795432 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84d5c7b66c-z8qvd" event={"ID":"5abd0e8b-5a4c-4117-be05-72d39fc316d6","Type":"ContainerDied","Data":"76b6379a97c95bff8c4d6e9560aacc28893411b91673a9f4ba5cfccc60a6cf8b"} Jul 27 20:49:47.795708 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:47.795481 2570 scope.go:117] "RemoveContainer" containerID="76b6379a97c95bff8c4d6e9560aacc28893411b91673a9f4ba5cfccc60a6cf8b" Jul 27 20:49:47.795708 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:47.795361 2570 generic.go:356] "Generic (PLEG): container finished" podID="5abd0e8b-5a4c-4117-be05-72d39fc316d6" containerID="76b6379a97c95bff8c4d6e9560aacc28893411b91673a9f4ba5cfccc60a6cf8b" exitCode=0 Jul 27 20:49:47.795708 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:47.795639 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84d5c7b66c-z8qvd" event={"ID":"5abd0e8b-5a4c-4117-be05-72d39fc316d6","Type":"ContainerDied","Data":"e3cfbc252ad26f024245818eaf904d8b0c50e6ad5f5a59bfc70f1fb4a91de3fd"} Jul 27 20:49:47.805034 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:47.805018 2570 scope.go:117] "RemoveContainer" containerID="76b6379a97c95bff8c4d6e9560aacc28893411b91673a9f4ba5cfccc60a6cf8b" Jul 27 20:49:47.805348 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:49:47.805327 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76b6379a97c95bff8c4d6e9560aacc28893411b91673a9f4ba5cfccc60a6cf8b\": container with ID starting with 76b6379a97c95bff8c4d6e9560aacc28893411b91673a9f4ba5cfccc60a6cf8b not found: ID does not exist" containerID="76b6379a97c95bff8c4d6e9560aacc28893411b91673a9f4ba5cfccc60a6cf8b" Jul 27 20:49:47.805428 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:47.805354 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76b6379a97c95bff8c4d6e9560aacc28893411b91673a9f4ba5cfccc60a6cf8b"} err="failed to get container status \"76b6379a97c95bff8c4d6e9560aacc28893411b91673a9f4ba5cfccc60a6cf8b\": rpc error: code = NotFound desc = could not find container \"76b6379a97c95bff8c4d6e9560aacc28893411b91673a9f4ba5cfccc60a6cf8b\": container with ID starting with 76b6379a97c95bff8c4d6e9560aacc28893411b91673a9f4ba5cfccc60a6cf8b not found: ID does not exist" Jul 27 20:49:47.820644 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:47.820611 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84d5c7b66c-z8qvd"] Jul 27 20:49:47.829361 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:47.829328 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-84d5c7b66c-z8qvd"] Jul 27 20:49:48.651864 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:48.651825 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84d5c7b66c-6ggpf"] Jul 27 20:49:48.652270 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:48.652258 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5abd0e8b-5a4c-4117-be05-72d39fc316d6" containerName="manager" Jul 27 20:49:48.652270 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:48.652271 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5abd0e8b-5a4c-4117-be05-72d39fc316d6" containerName="manager" Jul 27 20:49:48.652347 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:48.652342 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5abd0e8b-5a4c-4117-be05-72d39fc316d6" containerName="manager" Jul 27 20:49:48.655574 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:48.655556 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84d5c7b66c-6ggpf" Jul 27 20:49:48.658803 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:48.658781 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-chstr\"" Jul 27 20:49:48.658934 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:48.658783 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 27 20:49:48.658934 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:48.658788 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 27 20:49:48.665321 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:48.665294 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84d5c7b66c-6ggpf"] Jul 27 20:49:48.769713 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:48.769678 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c61fd4ad-ed54-40b1-ba3f-13d2ded266f8-webhook-cert\") pod \"maas-controller-84d5c7b66c-6ggpf\" (UID: \"c61fd4ad-ed54-40b1-ba3f-13d2ded266f8\") " pod="opendatahub/maas-controller-84d5c7b66c-6ggpf" Jul 27 20:49:48.769915 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:48.769744 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfrxq\" (UniqueName: \"kubernetes.io/projected/c61fd4ad-ed54-40b1-ba3f-13d2ded266f8-kube-api-access-hfrxq\") pod \"maas-controller-84d5c7b66c-6ggpf\" (UID: \"c61fd4ad-ed54-40b1-ba3f-13d2ded266f8\") " pod="opendatahub/maas-controller-84d5c7b66c-6ggpf" Jul 27 20:49:48.856817 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:48.855240 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5abd0e8b-5a4c-4117-be05-72d39fc316d6" path="/var/lib/kubelet/pods/5abd0e8b-5a4c-4117-be05-72d39fc316d6/volumes" Jul 27 20:49:48.870272 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:48.870233 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfrxq\" (UniqueName: \"kubernetes.io/projected/c61fd4ad-ed54-40b1-ba3f-13d2ded266f8-kube-api-access-hfrxq\") pod \"maas-controller-84d5c7b66c-6ggpf\" (UID: \"c61fd4ad-ed54-40b1-ba3f-13d2ded266f8\") " pod="opendatahub/maas-controller-84d5c7b66c-6ggpf" Jul 27 20:49:48.870461 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:48.870342 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c61fd4ad-ed54-40b1-ba3f-13d2ded266f8-webhook-cert\") pod \"maas-controller-84d5c7b66c-6ggpf\" (UID: \"c61fd4ad-ed54-40b1-ba3f-13d2ded266f8\") " pod="opendatahub/maas-controller-84d5c7b66c-6ggpf" Jul 27 20:49:48.873000 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:48.872978 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c61fd4ad-ed54-40b1-ba3f-13d2ded266f8-webhook-cert\") pod \"maas-controller-84d5c7b66c-6ggpf\" (UID: \"c61fd4ad-ed54-40b1-ba3f-13d2ded266f8\") " pod="opendatahub/maas-controller-84d5c7b66c-6ggpf" Jul 27 20:49:48.879194 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:48.879169 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfrxq\" (UniqueName: \"kubernetes.io/projected/c61fd4ad-ed54-40b1-ba3f-13d2ded266f8-kube-api-access-hfrxq\") pod \"maas-controller-84d5c7b66c-6ggpf\" (UID: \"c61fd4ad-ed54-40b1-ba3f-13d2ded266f8\") " pod="opendatahub/maas-controller-84d5c7b66c-6ggpf" Jul 27 20:49:48.967208 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:48.967181 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84d5c7b66c-6ggpf" Jul 27 20:49:49.109275 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:49.109250 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84d5c7b66c-6ggpf"] Jul 27 20:49:49.110952 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:49:49.110926 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc61fd4ad_ed54_40b1_ba3f_13d2ded266f8.slice/crio-337a8a16247eeff9c50981dfac48a84db35c5ee18b742731456aaf756b26604b WatchSource:0}: Error finding container 337a8a16247eeff9c50981dfac48a84db35c5ee18b742731456aaf756b26604b: Status 404 returned error can't find the container with id 337a8a16247eeff9c50981dfac48a84db35c5ee18b742731456aaf756b26604b Jul 27 20:49:49.112350 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:49.112334 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 20:49:49.810164 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:49.810122 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84d5c7b66c-6ggpf" event={"ID":"c61fd4ad-ed54-40b1-ba3f-13d2ded266f8","Type":"ContainerStarted","Data":"99cca9ff1d5304adcd22ac40c8af0dcd59fd2c48e5c1c0f3cbf6a54cbf791734"} Jul 27 20:49:49.810548 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:49.810197 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84d5c7b66c-6ggpf" Jul 27 20:49:49.810548 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:49.810225 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84d5c7b66c-6ggpf" event={"ID":"c61fd4ad-ed54-40b1-ba3f-13d2ded266f8","Type":"ContainerStarted","Data":"337a8a16247eeff9c50981dfac48a84db35c5ee18b742731456aaf756b26604b"} Jul 27 20:49:49.829155 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:49:49.829097 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84d5c7b66c-6ggpf" podStartSLOduration=1.518999508 podStartE2EDuration="1.82908102s" totalImagesPullingTime="310.081512ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:49:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:49:49.827554923 +0000 UTC m=+919.563535236" watchObservedRunningTime="2026-07-27 20:49:49.82908102 +0000 UTC m=+919.565061307" Jul 27 20:50:00.818417 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:50:00.818383 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84d5c7b66c-6ggpf" Jul 27 20:54:36.585856 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:54:36.585820 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6vzl"] Jul 27 20:54:36.586691 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:54:36.586664 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6vzl" podUID="11104d7e-28dc-48f7-afd9-de980be2d8f1" containerName="manager" containerID="cri-o://b90fa4a3b30a24ee4a595ee3ec8668db63897df3780b58f33bfbaa518962a979" gracePeriod=10 Jul 27 20:54:39.447943 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:54:39.447913 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6vzl" Jul 27 20:54:39.549158 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:54:39.549069 2570 generic.go:356] "Generic (PLEG): container finished" podID="11104d7e-28dc-48f7-afd9-de980be2d8f1" containerID="b90fa4a3b30a24ee4a595ee3ec8668db63897df3780b58f33bfbaa518962a979" exitCode=0 Jul 27 20:54:39.549158 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:54:39.549117 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6vzl" Jul 27 20:54:39.549372 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:54:39.549155 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6vzl" event={"ID":"11104d7e-28dc-48f7-afd9-de980be2d8f1","Type":"ContainerDied","Data":"b90fa4a3b30a24ee4a595ee3ec8668db63897df3780b58f33bfbaa518962a979"} Jul 27 20:54:39.549372 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:54:39.549210 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6vzl" event={"ID":"11104d7e-28dc-48f7-afd9-de980be2d8f1","Type":"ContainerDied","Data":"aaf5a225a97e7a59660ec1f2b48b81e0d00aec0a4b81202b931e225a867ab4e0"} Jul 27 20:54:39.549372 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:54:39.549238 2570 scope.go:117] "RemoveContainer" containerID="b90fa4a3b30a24ee4a595ee3ec8668db63897df3780b58f33bfbaa518962a979" Jul 27 20:54:39.553555 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:54:39.553531 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/11104d7e-28dc-48f7-afd9-de980be2d8f1-extensions-socket-volume\") pod \"11104d7e-28dc-48f7-afd9-de980be2d8f1\" (UID: \"11104d7e-28dc-48f7-afd9-de980be2d8f1\") " Jul 27 20:54:39.553709 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:54:39.553626 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvpkn\" (UniqueName: \"kubernetes.io/projected/11104d7e-28dc-48f7-afd9-de980be2d8f1-kube-api-access-zvpkn\") pod \"11104d7e-28dc-48f7-afd9-de980be2d8f1\" (UID: \"11104d7e-28dc-48f7-afd9-de980be2d8f1\") " Jul 27 20:54:39.553888 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:54:39.553866 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11104d7e-28dc-48f7-afd9-de980be2d8f1-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "11104d7e-28dc-48f7-afd9-de980be2d8f1" (UID: "11104d7e-28dc-48f7-afd9-de980be2d8f1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 20:54:39.553979 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:54:39.553964 2570 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/11104d7e-28dc-48f7-afd9-de980be2d8f1-extensions-socket-volume\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:54:39.555787 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:54:39.555764 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11104d7e-28dc-48f7-afd9-de980be2d8f1-kube-api-access-zvpkn" (OuterVolumeSpecName: "kube-api-access-zvpkn") pod "11104d7e-28dc-48f7-afd9-de980be2d8f1" (UID: "11104d7e-28dc-48f7-afd9-de980be2d8f1"). InnerVolumeSpecName "kube-api-access-zvpkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:54:39.561062 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:54:39.561041 2570 scope.go:117] "RemoveContainer" containerID="b90fa4a3b30a24ee4a595ee3ec8668db63897df3780b58f33bfbaa518962a979" Jul 27 20:54:39.561378 ip-10-0-137-141 kubenswrapper[2570]: E0727 20:54:39.561357 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b90fa4a3b30a24ee4a595ee3ec8668db63897df3780b58f33bfbaa518962a979\": container with ID starting with b90fa4a3b30a24ee4a595ee3ec8668db63897df3780b58f33bfbaa518962a979 not found: ID does not exist" containerID="b90fa4a3b30a24ee4a595ee3ec8668db63897df3780b58f33bfbaa518962a979" Jul 27 20:54:39.561445 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:54:39.561387 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b90fa4a3b30a24ee4a595ee3ec8668db63897df3780b58f33bfbaa518962a979"} err="failed to get container status \"b90fa4a3b30a24ee4a595ee3ec8668db63897df3780b58f33bfbaa518962a979\": rpc error: code = NotFound desc = could not find container \"b90fa4a3b30a24ee4a595ee3ec8668db63897df3780b58f33bfbaa518962a979\": container with ID starting with b90fa4a3b30a24ee4a595ee3ec8668db63897df3780b58f33bfbaa518962a979 not found: ID does not exist" Jul 27 20:54:39.655004 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:54:39.654966 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvpkn\" (UniqueName: \"kubernetes.io/projected/11104d7e-28dc-48f7-afd9-de980be2d8f1-kube-api-access-zvpkn\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 20:54:39.871287 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:54:39.871243 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6vzl"] Jul 27 20:54:39.874027 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:54:39.874000 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6vzl"] Jul 27 20:54:40.853796 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:54:40.853762 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11104d7e-28dc-48f7-afd9-de980be2d8f1" path="/var/lib/kubelet/pods/11104d7e-28dc-48f7-afd9-de980be2d8f1/volumes" Jul 27 20:55:37.729013 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:55:37.728978 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hkmk"] Jul 27 20:55:37.729546 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:55:37.729528 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11104d7e-28dc-48f7-afd9-de980be2d8f1" containerName="manager" Jul 27 20:55:37.729593 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:55:37.729549 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="11104d7e-28dc-48f7-afd9-de980be2d8f1" containerName="manager" Jul 27 20:55:37.729708 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:55:37.729697 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="11104d7e-28dc-48f7-afd9-de980be2d8f1" containerName="manager" Jul 27 20:55:37.733310 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:55:37.733295 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hkmk" Jul 27 20:55:37.738010 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:55:37.737983 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-2nshg\"" Jul 27 20:55:37.802547 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:55:37.802510 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6vhl\" (UniqueName: \"kubernetes.io/projected/253c0a6c-5f53-42e7-a2e7-c23b3c8c5702-kube-api-access-c6vhl\") pod \"kuadrant-operator-controller-manager-86878c769f-5hkmk\" (UID: \"253c0a6c-5f53-42e7-a2e7-c23b3c8c5702\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hkmk" Jul 27 20:55:37.802734 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:55:37.802566 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/253c0a6c-5f53-42e7-a2e7-c23b3c8c5702-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5hkmk\" (UID: \"253c0a6c-5f53-42e7-a2e7-c23b3c8c5702\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hkmk" Jul 27 20:55:37.825093 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:55:37.825056 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hkmk"] Jul 27 20:55:37.903811 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:55:37.903776 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6vhl\" (UniqueName: \"kubernetes.io/projected/253c0a6c-5f53-42e7-a2e7-c23b3c8c5702-kube-api-access-c6vhl\") pod \"kuadrant-operator-controller-manager-86878c769f-5hkmk\" (UID: \"253c0a6c-5f53-42e7-a2e7-c23b3c8c5702\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hkmk" Jul 27 20:55:37.904037 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:55:37.903839 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/253c0a6c-5f53-42e7-a2e7-c23b3c8c5702-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5hkmk\" (UID: \"253c0a6c-5f53-42e7-a2e7-c23b3c8c5702\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hkmk" Jul 27 20:55:37.904246 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:55:37.904219 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/253c0a6c-5f53-42e7-a2e7-c23b3c8c5702-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5hkmk\" (UID: \"253c0a6c-5f53-42e7-a2e7-c23b3c8c5702\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hkmk" Jul 27 20:55:37.950521 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:55:37.950489 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6vhl\" (UniqueName: \"kubernetes.io/projected/253c0a6c-5f53-42e7-a2e7-c23b3c8c5702-kube-api-access-c6vhl\") pod \"kuadrant-operator-controller-manager-86878c769f-5hkmk\" (UID: \"253c0a6c-5f53-42e7-a2e7-c23b3c8c5702\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hkmk" Jul 27 20:55:38.043814 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:55:38.043712 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hkmk" Jul 27 20:55:38.198345 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:55:38.198317 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hkmk"] Jul 27 20:55:38.199214 ip-10-0-137-141 kubenswrapper[2570]: W0727 20:55:38.199190 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod253c0a6c_5f53_42e7_a2e7_c23b3c8c5702.slice/crio-08cc3ba4a8a0da0fea77916252bfe7c385bfbbe8d968ef81b11dfcd178fea729 WatchSource:0}: Error finding container 08cc3ba4a8a0da0fea77916252bfe7c385bfbbe8d968ef81b11dfcd178fea729: Status 404 returned error can't find the container with id 08cc3ba4a8a0da0fea77916252bfe7c385bfbbe8d968ef81b11dfcd178fea729 Jul 27 20:55:38.905208 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:55:38.905171 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hkmk" event={"ID":"253c0a6c-5f53-42e7-a2e7-c23b3c8c5702","Type":"ContainerStarted","Data":"e4c70fea50205049096777745928a8fc87e7acf165fc3b09c11d315d35d69173"} Jul 27 20:55:38.905622 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:55:38.905236 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hkmk" event={"ID":"253c0a6c-5f53-42e7-a2e7-c23b3c8c5702","Type":"ContainerStarted","Data":"08cc3ba4a8a0da0fea77916252bfe7c385bfbbe8d968ef81b11dfcd178fea729"} Jul 27 20:55:38.905622 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:55:38.905404 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hkmk" Jul 27 20:55:38.927738 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:55:38.927690 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hkmk" podStartSLOduration=1.927672033 podStartE2EDuration="1.927672033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:55:38.925808532 +0000 UTC m=+1268.661788818" watchObservedRunningTime="2026-07-27 20:55:38.927672033 +0000 UTC m=+1268.663652320" Jul 27 20:55:49.913517 ip-10-0-137-141 kubenswrapper[2570]: I0727 20:55:49.913479 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hkmk" Jul 27 21:00:00.137595 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:00:00.137508 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753100-bsfpt"] Jul 27 21:00:00.141228 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:00:00.141204 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753100-bsfpt" Jul 27 21:00:00.143952 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:00:00.143928 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jzmp2\"" Jul 27 21:00:00.147564 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:00:00.147537 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753100-bsfpt"] Jul 27 21:00:00.153391 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:00:00.153366 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753100-bsfpt" Jul 27 21:00:00.301008 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:00:00.300978 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753100-bsfpt"] Jul 27 21:00:00.303236 ip-10-0-137-141 kubenswrapper[2570]: W0727 21:00:00.303206 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab72247f_df8e_435e_bdee_941ae990cceb.slice/crio-69c9e949f3c11e4a20b0eed8d88d12337f93f843e37f81195ddbc0aa6181cba1 WatchSource:0}: Error finding container 69c9e949f3c11e4a20b0eed8d88d12337f93f843e37f81195ddbc0aa6181cba1: Status 404 returned error can't find the container with id 69c9e949f3c11e4a20b0eed8d88d12337f93f843e37f81195ddbc0aa6181cba1 Jul 27 21:00:00.305206 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:00:00.305180 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 21:00:00.477140 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:00:00.477104 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753100-bsfpt" event={"ID":"ab72247f-df8e-435e-bdee-941ae990cceb","Type":"ContainerStarted","Data":"69c9e949f3c11e4a20b0eed8d88d12337f93f843e37f81195ddbc0aa6181cba1"} Jul 27 21:00:05.511221 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:00:05.511179 2570 generic.go:356] "Generic (PLEG): container finished" podID="ab72247f-df8e-435e-bdee-941ae990cceb" containerID="145fc43645a4eaf9cb55f1952cae5e50865e25a2be17c4a5dfd322aff4021a4c" exitCode=0 Jul 27 21:00:05.511718 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:00:05.511258 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753100-bsfpt" event={"ID":"ab72247f-df8e-435e-bdee-941ae990cceb","Type":"ContainerDied","Data":"145fc43645a4eaf9cb55f1952cae5e50865e25a2be17c4a5dfd322aff4021a4c"} Jul 27 21:00:06.666881 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:00:06.666850 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753100-bsfpt" Jul 27 21:00:07.524349 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:00:07.524323 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753100-bsfpt" Jul 27 21:00:07.524530 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:00:07.524324 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753100-bsfpt" event={"ID":"ab72247f-df8e-435e-bdee-941ae990cceb","Type":"ContainerDied","Data":"69c9e949f3c11e4a20b0eed8d88d12337f93f843e37f81195ddbc0aa6181cba1"} Jul 27 21:00:07.524530 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:00:07.524440 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69c9e949f3c11e4a20b0eed8d88d12337f93f843e37f81195ddbc0aa6181cba1" Jul 27 21:02:57.354052 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:02:57.353967 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs"] Jul 27 21:02:57.354866 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:02:57.354525 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab72247f-df8e-435e-bdee-941ae990cceb" containerName="cleanup" Jul 27 21:02:57.354866 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:02:57.354540 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab72247f-df8e-435e-bdee-941ae990cceb" containerName="cleanup" Jul 27 21:02:57.354866 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:02:57.354676 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab72247f-df8e-435e-bdee-941ae990cceb" containerName="cleanup" Jul 27 21:02:57.357020 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:02:57.357001 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs" Jul 27 21:02:57.360095 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:02:57.360073 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f9e4b23a-serving-cert\"" Jul 27 21:02:57.360446 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:02:57.360429 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cvmwf\"" Jul 27 21:02:57.378137 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:02:57.378108 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs"] Jul 27 21:02:57.459103 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:02:57.459057 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec275f62-c7be-41bd-95a9-b6e99bcad8af-maas-api-tls\") pod \"maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs\" (UID: \"ec275f62-c7be-41bd-95a9-b6e99bcad8af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs" Jul 27 21:02:57.459292 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:02:57.459119 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw4hj\" (UniqueName: \"kubernetes.io/projected/ec275f62-c7be-41bd-95a9-b6e99bcad8af-kube-api-access-mw4hj\") pod \"maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs\" (UID: \"ec275f62-c7be-41bd-95a9-b6e99bcad8af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs" Jul 27 21:02:57.459292 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:02:57.459187 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec275f62-c7be-41bd-95a9-b6e99bcad8af-openshift-service-ca\") pod \"maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs\" (UID: \"ec275f62-c7be-41bd-95a9-b6e99bcad8af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs" Jul 27 21:02:57.559917 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:02:57.559879 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec275f62-c7be-41bd-95a9-b6e99bcad8af-openshift-service-ca\") pod \"maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs\" (UID: \"ec275f62-c7be-41bd-95a9-b6e99bcad8af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs" Jul 27 21:02:57.560130 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:02:57.560033 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec275f62-c7be-41bd-95a9-b6e99bcad8af-maas-api-tls\") pod \"maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs\" (UID: \"ec275f62-c7be-41bd-95a9-b6e99bcad8af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs" Jul 27 21:02:57.560130 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:02:57.560072 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw4hj\" (UniqueName: \"kubernetes.io/projected/ec275f62-c7be-41bd-95a9-b6e99bcad8af-kube-api-access-mw4hj\") pod \"maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs\" (UID: \"ec275f62-c7be-41bd-95a9-b6e99bcad8af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs" Jul 27 21:02:57.560253 ip-10-0-137-141 kubenswrapper[2570]: E0727 21:02:57.560181 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f9e4b23a-serving-cert: secret "maas-api-e2e-ait-f9e4b23a-serving-cert" not found Jul 27 21:02:57.560253 ip-10-0-137-141 kubenswrapper[2570]: E0727 21:02:57.560249 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec275f62-c7be-41bd-95a9-b6e99bcad8af-maas-api-tls podName:ec275f62-c7be-41bd-95a9-b6e99bcad8af nodeName:}" failed. No retries permitted until 2026-07-27 21:02:58.060228534 +0000 UTC m=+1707.796208803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ec275f62-c7be-41bd-95a9-b6e99bcad8af-maas-api-tls") pod "maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs" (UID: "ec275f62-c7be-41bd-95a9-b6e99bcad8af") : secret "maas-api-e2e-ait-f9e4b23a-serving-cert" not found Jul 27 21:02:57.560657 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:02:57.560636 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec275f62-c7be-41bd-95a9-b6e99bcad8af-openshift-service-ca\") pod \"maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs\" (UID: \"ec275f62-c7be-41bd-95a9-b6e99bcad8af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs" Jul 27 21:02:57.570185 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:02:57.570158 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw4hj\" (UniqueName: \"kubernetes.io/projected/ec275f62-c7be-41bd-95a9-b6e99bcad8af-kube-api-access-mw4hj\") pod \"maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs\" (UID: \"ec275f62-c7be-41bd-95a9-b6e99bcad8af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs" Jul 27 21:02:58.065364 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:02:58.065330 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec275f62-c7be-41bd-95a9-b6e99bcad8af-maas-api-tls\") pod \"maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs\" (UID: \"ec275f62-c7be-41bd-95a9-b6e99bcad8af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs" Jul 27 21:02:58.068050 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:02:58.067994 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec275f62-c7be-41bd-95a9-b6e99bcad8af-maas-api-tls\") pod \"maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs\" (UID: \"ec275f62-c7be-41bd-95a9-b6e99bcad8af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs" Jul 27 21:02:58.267464 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:02:58.267421 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs" Jul 27 21:02:58.434439 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:02:58.434407 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs"] Jul 27 21:02:58.435663 ip-10-0-137-141 kubenswrapper[2570]: W0727 21:02:58.435633 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec275f62_c7be_41bd_95a9_b6e99bcad8af.slice/crio-bba90fffda1a6c7555cb3a9dfe16f034950f24844554d143789621a189aec3f6 WatchSource:0}: Error finding container bba90fffda1a6c7555cb3a9dfe16f034950f24844554d143789621a189aec3f6: Status 404 returned error can't find the container with id bba90fffda1a6c7555cb3a9dfe16f034950f24844554d143789621a189aec3f6 Jul 27 21:02:58.571132 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:02:58.571087 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs" event={"ID":"ec275f62-c7be-41bd-95a9-b6e99bcad8af","Type":"ContainerStarted","Data":"bba90fffda1a6c7555cb3a9dfe16f034950f24844554d143789621a189aec3f6"} Jul 27 21:02:59.588704 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:02:59.588660 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs" event={"ID":"ec275f62-c7be-41bd-95a9-b6e99bcad8af","Type":"ContainerStarted","Data":"547d4774129d4eaf170c27f982f71dafa49a95f0ef4fbd40da6a16ed812c87a1"} Jul 27 21:02:59.589159 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:02:59.588896 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs" Jul 27 21:02:59.607479 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:02:59.607412 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs" podStartSLOduration=1.905245676 podStartE2EDuration="2.60739743s" totalImagesPullingTime="702.151754ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:02:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:02:59.60491891 +0000 UTC m=+1709.340899198" watchObservedRunningTime="2026-07-27 21:02:59.60739743 +0000 UTC m=+1709.343377716" Jul 27 21:03:05.600298 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:05.600265 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs" Jul 27 21:03:22.789264 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:22.789213 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e4b23a-lzfgp"] Jul 27 21:03:22.792191 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:22.792169 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e4b23a-lzfgp" Jul 27 21:03:22.794721 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:22.794698 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jzmp2\"" Jul 27 21:03:22.800852 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:22.800484 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e4b23a-lzfgp"] Jul 27 21:03:22.907018 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:22.906981 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eacff56d-f697-40ca-a31c-a374b096306f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9e4b23a-lzfgp\" (UID: \"eacff56d-f697-40ca-a31c-a374b096306f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e4b23a-lzfgp" Jul 27 21:03:23.008577 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:23.008538 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eacff56d-f697-40ca-a31c-a374b096306f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9e4b23a-lzfgp\" (UID: \"eacff56d-f697-40ca-a31c-a374b096306f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e4b23a-lzfgp" Jul 27 21:03:23.009221 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:23.009200 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eacff56d-f697-40ca-a31c-a374b096306f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9e4b23a-lzfgp\" (UID: \"eacff56d-f697-40ca-a31c-a374b096306f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e4b23a-lzfgp" Jul 27 21:03:23.105534 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:23.105449 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e4b23a-lzfgp" Jul 27 21:03:23.264058 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:23.264026 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e4b23a-lzfgp"] Jul 27 21:03:23.264886 ip-10-0-137-141 kubenswrapper[2570]: W0727 21:03:23.264840 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeacff56d_f697_40ca_a31c_a374b096306f.slice/crio-e437716e83ed37420d1de656b70a6d94ca6a91043737e3aaa62780f29c0e5eb9 WatchSource:0}: Error finding container e437716e83ed37420d1de656b70a6d94ca6a91043737e3aaa62780f29c0e5eb9: Status 404 returned error can't find the container with id e437716e83ed37420d1de656b70a6d94ca6a91043737e3aaa62780f29c0e5eb9 Jul 27 21:03:23.732267 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:23.732234 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e4b23a-lzfgp" event={"ID":"eacff56d-f697-40ca-a31c-a374b096306f","Type":"ContainerStarted","Data":"6cab54eff0fd66848b9a34ccfd6b25685eab8917fe0e20294a3e84b6058558ee"} Jul 27 21:03:23.732456 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:23.732290 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e4b23a-lzfgp" event={"ID":"eacff56d-f697-40ca-a31c-a374b096306f","Type":"ContainerStarted","Data":"e437716e83ed37420d1de656b70a6d94ca6a91043737e3aaa62780f29c0e5eb9"} Jul 27 21:03:23.750984 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:23.750933 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e4b23a-lzfgp" podStartSLOduration=1.750916825 podStartE2EDuration="1.750916825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:03:23.748769435 +0000 UTC m=+1733.484749725" watchObservedRunningTime="2026-07-27 21:03:23.750916825 +0000 UTC m=+1733.486897111" Jul 27 21:03:24.739146 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:24.739109 2570 generic.go:356] "Generic (PLEG): container finished" podID="eacff56d-f697-40ca-a31c-a374b096306f" containerID="6cab54eff0fd66848b9a34ccfd6b25685eab8917fe0e20294a3e84b6058558ee" exitCode=0 Jul 27 21:03:24.739648 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:24.739167 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e4b23a-lzfgp" event={"ID":"eacff56d-f697-40ca-a31c-a374b096306f","Type":"ContainerDied","Data":"6cab54eff0fd66848b9a34ccfd6b25685eab8917fe0e20294a3e84b6058558ee"} Jul 27 21:03:25.886948 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:25.886924 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e4b23a-lzfgp" Jul 27 21:03:25.939859 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:25.939826 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eacff56d-f697-40ca-a31c-a374b096306f-maas-api-service-ca\") pod \"eacff56d-f697-40ca-a31c-a374b096306f\" (UID: \"eacff56d-f697-40ca-a31c-a374b096306f\") " Jul 27 21:03:25.940200 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:25.940172 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eacff56d-f697-40ca-a31c-a374b096306f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eacff56d-f697-40ca-a31c-a374b096306f" (UID: "eacff56d-f697-40ca-a31c-a374b096306f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:03:26.041384 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:26.041299 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eacff56d-f697-40ca-a31c-a374b096306f-maas-api-service-ca\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 21:03:26.752289 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:26.752258 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e4b23a-lzfgp" Jul 27 21:03:26.752470 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:26.752297 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e4b23a-lzfgp" event={"ID":"eacff56d-f697-40ca-a31c-a374b096306f","Type":"ContainerDied","Data":"e437716e83ed37420d1de656b70a6d94ca6a91043737e3aaa62780f29c0e5eb9"} Jul 27 21:03:26.752470 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:26.752343 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e437716e83ed37420d1de656b70a6d94ca6a91043737e3aaa62780f29c0e5eb9" Jul 27 21:03:33.228563 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:33.228533 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs"] Jul 27 21:03:33.229168 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:33.228804 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs" podUID="ec275f62-c7be-41bd-95a9-b6e99bcad8af" containerName="maas-api" containerID="cri-o://547d4774129d4eaf170c27f982f71dafa49a95f0ef4fbd40da6a16ed812c87a1" gracePeriod=30 Jul 27 21:03:33.495579 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:33.495552 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs" Jul 27 21:03:33.613442 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:33.613407 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec275f62-c7be-41bd-95a9-b6e99bcad8af-openshift-service-ca\") pod \"ec275f62-c7be-41bd-95a9-b6e99bcad8af\" (UID: \"ec275f62-c7be-41bd-95a9-b6e99bcad8af\") " Jul 27 21:03:33.613641 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:33.613501 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec275f62-c7be-41bd-95a9-b6e99bcad8af-maas-api-tls\") pod \"ec275f62-c7be-41bd-95a9-b6e99bcad8af\" (UID: \"ec275f62-c7be-41bd-95a9-b6e99bcad8af\") " Jul 27 21:03:33.613641 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:33.613535 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mw4hj\" (UniqueName: \"kubernetes.io/projected/ec275f62-c7be-41bd-95a9-b6e99bcad8af-kube-api-access-mw4hj\") pod \"ec275f62-c7be-41bd-95a9-b6e99bcad8af\" (UID: \"ec275f62-c7be-41bd-95a9-b6e99bcad8af\") " Jul 27 21:03:33.613857 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:33.613829 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec275f62-c7be-41bd-95a9-b6e99bcad8af-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ec275f62-c7be-41bd-95a9-b6e99bcad8af" (UID: "ec275f62-c7be-41bd-95a9-b6e99bcad8af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:03:33.615993 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:33.615958 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec275f62-c7be-41bd-95a9-b6e99bcad8af-kube-api-access-mw4hj" (OuterVolumeSpecName: "kube-api-access-mw4hj") pod "ec275f62-c7be-41bd-95a9-b6e99bcad8af" (UID: "ec275f62-c7be-41bd-95a9-b6e99bcad8af"). InnerVolumeSpecName "kube-api-access-mw4hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:03:33.616109 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:33.616053 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec275f62-c7be-41bd-95a9-b6e99bcad8af-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ec275f62-c7be-41bd-95a9-b6e99bcad8af" (UID: "ec275f62-c7be-41bd-95a9-b6e99bcad8af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 21:03:33.715266 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:33.715223 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec275f62-c7be-41bd-95a9-b6e99bcad8af-openshift-service-ca\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 21:03:33.715266 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:33.715258 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec275f62-c7be-41bd-95a9-b6e99bcad8af-maas-api-tls\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 21:03:33.715266 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:33.715268 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw4hj\" (UniqueName: \"kubernetes.io/projected/ec275f62-c7be-41bd-95a9-b6e99bcad8af-kube-api-access-mw4hj\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 21:03:33.796458 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:33.796361 2570 generic.go:356] "Generic (PLEG): container finished" podID="ec275f62-c7be-41bd-95a9-b6e99bcad8af" containerID="547d4774129d4eaf170c27f982f71dafa49a95f0ef4fbd40da6a16ed812c87a1" exitCode=0 Jul 27 21:03:33.796458 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:33.796437 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs" Jul 27 21:03:33.796458 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:33.796442 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs" event={"ID":"ec275f62-c7be-41bd-95a9-b6e99bcad8af","Type":"ContainerDied","Data":"547d4774129d4eaf170c27f982f71dafa49a95f0ef4fbd40da6a16ed812c87a1"} Jul 27 21:03:33.796708 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:33.796522 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs" event={"ID":"ec275f62-c7be-41bd-95a9-b6e99bcad8af","Type":"ContainerDied","Data":"bba90fffda1a6c7555cb3a9dfe16f034950f24844554d143789621a189aec3f6"} Jul 27 21:03:33.796708 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:33.796545 2570 scope.go:117] "RemoveContainer" containerID="547d4774129d4eaf170c27f982f71dafa49a95f0ef4fbd40da6a16ed812c87a1" Jul 27 21:03:33.806872 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:33.806850 2570 scope.go:117] "RemoveContainer" containerID="547d4774129d4eaf170c27f982f71dafa49a95f0ef4fbd40da6a16ed812c87a1" Jul 27 21:03:33.807130 ip-10-0-137-141 kubenswrapper[2570]: E0727 21:03:33.807112 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"547d4774129d4eaf170c27f982f71dafa49a95f0ef4fbd40da6a16ed812c87a1\": container with ID starting with 547d4774129d4eaf170c27f982f71dafa49a95f0ef4fbd40da6a16ed812c87a1 not found: ID does not exist" containerID="547d4774129d4eaf170c27f982f71dafa49a95f0ef4fbd40da6a16ed812c87a1" Jul 27 21:03:33.807178 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:33.807140 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"547d4774129d4eaf170c27f982f71dafa49a95f0ef4fbd40da6a16ed812c87a1"} err="failed to get container status \"547d4774129d4eaf170c27f982f71dafa49a95f0ef4fbd40da6a16ed812c87a1\": rpc error: code = NotFound desc = could not find container \"547d4774129d4eaf170c27f982f71dafa49a95f0ef4fbd40da6a16ed812c87a1\": container with ID starting with 547d4774129d4eaf170c27f982f71dafa49a95f0ef4fbd40da6a16ed812c87a1 not found: ID does not exist" Jul 27 21:03:33.818786 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:33.818758 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs"] Jul 27 21:03:33.822753 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:33.822720 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f9e4b23a-57868bb7fd-d8gzs"] Jul 27 21:03:34.855144 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:34.855108 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec275f62-c7be-41bd-95a9-b6e99bcad8af" path="/var/lib/kubelet/pods/ec275f62-c7be-41bd-95a9-b6e99bcad8af/volumes" Jul 27 21:03:37.866260 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:37.866224 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e4b23a-lzfgp"] Jul 27 21:03:37.869399 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:37.869368 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9e4b23a-lzfgp"] Jul 27 21:03:38.854301 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:38.854265 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eacff56d-f697-40ca-a31c-a374b096306f" path="/var/lib/kubelet/pods/eacff56d-f697-40ca-a31c-a374b096306f/volumes" Jul 27 21:03:41.548368 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:41.548333 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn"] Jul 27 21:03:41.548813 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:41.548800 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eacff56d-f697-40ca-a31c-a374b096306f" containerName="revoke-keys" Jul 27 21:03:41.548859 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:41.548815 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="eacff56d-f697-40ca-a31c-a374b096306f" containerName="revoke-keys" Jul 27 21:03:41.548859 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:41.548848 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec275f62-c7be-41bd-95a9-b6e99bcad8af" containerName="maas-api" Jul 27 21:03:41.548859 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:41.548854 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec275f62-c7be-41bd-95a9-b6e99bcad8af" containerName="maas-api" Jul 27 21:03:41.548953 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:41.548917 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec275f62-c7be-41bd-95a9-b6e99bcad8af" containerName="maas-api" Jul 27 21:03:41.548953 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:41.548927 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="eacff56d-f697-40ca-a31c-a374b096306f" containerName="revoke-keys" Jul 27 21:03:41.551947 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:41.551928 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn" Jul 27 21:03:41.554752 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:41.554725 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cvmwf\"" Jul 27 21:03:41.554891 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:41.554804 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-4d7f566d-serving-cert\"" Jul 27 21:03:41.562376 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:41.562351 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn"] Jul 27 21:03:41.695034 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:41.694997 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be0575a0-04d2-41c2-8f86-e8b6f1f2e112-maas-api-tls\") pod \"maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn\" (UID: \"be0575a0-04d2-41c2-8f86-e8b6f1f2e112\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn" Jul 27 21:03:41.695233 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:41.695072 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpbjm\" (UniqueName: \"kubernetes.io/projected/be0575a0-04d2-41c2-8f86-e8b6f1f2e112-kube-api-access-qpbjm\") pod \"maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn\" (UID: \"be0575a0-04d2-41c2-8f86-e8b6f1f2e112\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn" Jul 27 21:03:41.695279 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:41.695232 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be0575a0-04d2-41c2-8f86-e8b6f1f2e112-openshift-service-ca\") pod \"maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn\" (UID: \"be0575a0-04d2-41c2-8f86-e8b6f1f2e112\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn" Jul 27 21:03:41.796247 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:41.796204 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be0575a0-04d2-41c2-8f86-e8b6f1f2e112-openshift-service-ca\") pod \"maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn\" (UID: \"be0575a0-04d2-41c2-8f86-e8b6f1f2e112\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn" Jul 27 21:03:41.796407 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:41.796258 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be0575a0-04d2-41c2-8f86-e8b6f1f2e112-maas-api-tls\") pod \"maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn\" (UID: \"be0575a0-04d2-41c2-8f86-e8b6f1f2e112\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn" Jul 27 21:03:41.796407 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:41.796297 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpbjm\" (UniqueName: \"kubernetes.io/projected/be0575a0-04d2-41c2-8f86-e8b6f1f2e112-kube-api-access-qpbjm\") pod \"maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn\" (UID: \"be0575a0-04d2-41c2-8f86-e8b6f1f2e112\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn" Jul 27 21:03:41.796407 ip-10-0-137-141 kubenswrapper[2570]: E0727 21:03:41.796391 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-4d7f566d-serving-cert: secret "maas-api-e2e-migrate-4d7f566d-serving-cert" not found Jul 27 21:03:41.796555 ip-10-0-137-141 kubenswrapper[2570]: E0727 21:03:41.796464 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be0575a0-04d2-41c2-8f86-e8b6f1f2e112-maas-api-tls podName:be0575a0-04d2-41c2-8f86-e8b6f1f2e112 nodeName:}" failed. No retries permitted until 2026-07-27 21:03:42.296443285 +0000 UTC m=+1752.032423551 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/be0575a0-04d2-41c2-8f86-e8b6f1f2e112-maas-api-tls") pod "maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn" (UID: "be0575a0-04d2-41c2-8f86-e8b6f1f2e112") : secret "maas-api-e2e-migrate-4d7f566d-serving-cert" not found Jul 27 21:03:41.797074 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:41.797048 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be0575a0-04d2-41c2-8f86-e8b6f1f2e112-openshift-service-ca\") pod \"maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn\" (UID: \"be0575a0-04d2-41c2-8f86-e8b6f1f2e112\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn" Jul 27 21:03:41.807792 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:41.807734 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpbjm\" (UniqueName: \"kubernetes.io/projected/be0575a0-04d2-41c2-8f86-e8b6f1f2e112-kube-api-access-qpbjm\") pod \"maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn\" (UID: \"be0575a0-04d2-41c2-8f86-e8b6f1f2e112\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn" Jul 27 21:03:42.300964 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:42.300918 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be0575a0-04d2-41c2-8f86-e8b6f1f2e112-maas-api-tls\") pod \"maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn\" (UID: \"be0575a0-04d2-41c2-8f86-e8b6f1f2e112\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn" Jul 27 21:03:42.303924 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:42.303893 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be0575a0-04d2-41c2-8f86-e8b6f1f2e112-maas-api-tls\") pod \"maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn\" (UID: \"be0575a0-04d2-41c2-8f86-e8b6f1f2e112\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn" Jul 27 21:03:42.418421 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:42.418380 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4d7f566d-7c8df99c5c-6v4t6"] Jul 27 21:03:42.421366 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:42.421338 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4d7f566d-7c8df99c5c-6v4t6" Jul 27 21:03:42.424051 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:42.424023 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-4d7f566d-dockercfg-qvvgw\"" Jul 27 21:03:42.424217 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:42.424060 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-4d7f566d\"" Jul 27 21:03:42.433958 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:42.433920 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4d7f566d-7c8df99c5c-6v4t6"] Jul 27 21:03:42.466538 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:42.466500 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn" Jul 27 21:03:42.604550 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:42.604515 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8hcn\" (UniqueName: \"kubernetes.io/projected/07615656-d386-477b-81c8-75786ff768c6-kube-api-access-h8hcn\") pod \"payload-processing-e2e-migrate-4d7f566d-7c8df99c5c-6v4t6\" (UID: \"07615656-d386-477b-81c8-75786ff768c6\") " pod="openshift-ingress/payload-processing-e2e-migrate-4d7f566d-7c8df99c5c-6v4t6" Jul 27 21:03:42.604980 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:42.604573 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07615656-d386-477b-81c8-75786ff768c6-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4d7f566d-7c8df99c5c-6v4t6\" (UID: \"07615656-d386-477b-81c8-75786ff768c6\") " pod="openshift-ingress/payload-processing-e2e-migrate-4d7f566d-7c8df99c5c-6v4t6" Jul 27 21:03:42.630127 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:42.630100 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn"] Jul 27 21:03:42.632067 ip-10-0-137-141 kubenswrapper[2570]: W0727 21:03:42.632037 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe0575a0_04d2_41c2_8f86_e8b6f1f2e112.slice/crio-01842a605fe5b256a96c6d5cb5b1ee91de045ffae88d6e9181961b069b28c836 WatchSource:0}: Error finding container 01842a605fe5b256a96c6d5cb5b1ee91de045ffae88d6e9181961b069b28c836: Status 404 returned error can't find the container with id 01842a605fe5b256a96c6d5cb5b1ee91de045ffae88d6e9181961b069b28c836 Jul 27 21:03:42.706024 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:42.705967 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8hcn\" (UniqueName: \"kubernetes.io/projected/07615656-d386-477b-81c8-75786ff768c6-kube-api-access-h8hcn\") pod \"payload-processing-e2e-migrate-4d7f566d-7c8df99c5c-6v4t6\" (UID: \"07615656-d386-477b-81c8-75786ff768c6\") " pod="openshift-ingress/payload-processing-e2e-migrate-4d7f566d-7c8df99c5c-6v4t6" Jul 27 21:03:42.706226 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:42.706057 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07615656-d386-477b-81c8-75786ff768c6-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4d7f566d-7c8df99c5c-6v4t6\" (UID: \"07615656-d386-477b-81c8-75786ff768c6\") " pod="openshift-ingress/payload-processing-e2e-migrate-4d7f566d-7c8df99c5c-6v4t6" Jul 27 21:03:42.706820 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:42.706802 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07615656-d386-477b-81c8-75786ff768c6-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4d7f566d-7c8df99c5c-6v4t6\" (UID: \"07615656-d386-477b-81c8-75786ff768c6\") " pod="openshift-ingress/payload-processing-e2e-migrate-4d7f566d-7c8df99c5c-6v4t6" Jul 27 21:03:42.714702 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:42.714678 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8hcn\" (UniqueName: \"kubernetes.io/projected/07615656-d386-477b-81c8-75786ff768c6-kube-api-access-h8hcn\") pod \"payload-processing-e2e-migrate-4d7f566d-7c8df99c5c-6v4t6\" (UID: \"07615656-d386-477b-81c8-75786ff768c6\") " pod="openshift-ingress/payload-processing-e2e-migrate-4d7f566d-7c8df99c5c-6v4t6" Jul 27 21:03:42.735269 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:42.735224 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4d7f566d-7c8df99c5c-6v4t6" Jul 27 21:03:42.857460 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:42.857406 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn" event={"ID":"be0575a0-04d2-41c2-8f86-e8b6f1f2e112","Type":"ContainerStarted","Data":"01842a605fe5b256a96c6d5cb5b1ee91de045ffae88d6e9181961b069b28c836"} Jul 27 21:03:42.885513 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:42.885485 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4d7f566d-7c8df99c5c-6v4t6"] Jul 27 21:03:42.887991 ip-10-0-137-141 kubenswrapper[2570]: W0727 21:03:42.887959 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07615656_d386_477b_81c8_75786ff768c6.slice/crio-50b7b57534a92c7a756f2cf162b7819dafbb2b7a89ccaf99fa71b960703543a3 WatchSource:0}: Error finding container 50b7b57534a92c7a756f2cf162b7819dafbb2b7a89ccaf99fa71b960703543a3: Status 404 returned error can't find the container with id 50b7b57534a92c7a756f2cf162b7819dafbb2b7a89ccaf99fa71b960703543a3 Jul 27 21:03:43.869487 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:43.869418 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4d7f566d-7c8df99c5c-6v4t6" event={"ID":"07615656-d386-477b-81c8-75786ff768c6","Type":"ContainerStarted","Data":"2c5855e0a3f3b47f10319935bc6de9e8189bfa7860fe14923cfd479df7903c74"} Jul 27 21:03:43.870088 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:43.869527 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4d7f566d-7c8df99c5c-6v4t6" event={"ID":"07615656-d386-477b-81c8-75786ff768c6","Type":"ContainerStarted","Data":"50b7b57534a92c7a756f2cf162b7819dafbb2b7a89ccaf99fa71b960703543a3"} Jul 27 21:03:43.870088 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:43.869837 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-4d7f566d-7c8df99c5c-6v4t6" Jul 27 21:03:43.871451 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:43.871425 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn" event={"ID":"be0575a0-04d2-41c2-8f86-e8b6f1f2e112","Type":"ContainerStarted","Data":"f55375a03e0a04c2601a14efbcea16c1de12f2902e782d4166b85444a88eb629"} Jul 27 21:03:43.871651 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:43.871633 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn" Jul 27 21:03:43.890352 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:43.890300 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-4d7f566d-7c8df99c5c-6v4t6" podStartSLOduration=1.890284804 podStartE2EDuration="1.890284804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:03:43.885912368 +0000 UTC m=+1753.621892659" watchObservedRunningTime="2026-07-27 21:03:43.890284804 +0000 UTC m=+1753.626265091" Jul 27 21:03:43.905063 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:43.905007 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn" podStartSLOduration=2.178243194 podStartE2EDuration="2.90499084s" totalImagesPullingTime="726.747646ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:03:43.90271917 +0000 UTC m=+1753.638699459" watchObservedRunningTime="2026-07-27 21:03:43.90499084 +0000 UTC m=+1753.640971127" Jul 27 21:03:49.883545 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:49.883516 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn" Jul 27 21:03:54.880759 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:03:54.880715 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-4d7f566d-7c8df99c5c-6v4t6" Jul 27 21:04:06.488314 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:06.488275 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d7f566d-64nxd"] Jul 27 21:04:06.491238 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:06.491220 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d7f566d-64nxd" Jul 27 21:04:06.493685 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:06.493665 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jzmp2\"" Jul 27 21:04:06.499302 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:06.499277 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d7f566d-64nxd"] Jul 27 21:04:06.639581 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:06.639547 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2307554c-7f3e-4af1-bcbd-cff6abd178aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d7f566d-64nxd\" (UID: \"2307554c-7f3e-4af1-bcbd-cff6abd178aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d7f566d-64nxd" Jul 27 21:04:06.741099 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:06.741006 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2307554c-7f3e-4af1-bcbd-cff6abd178aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d7f566d-64nxd\" (UID: \"2307554c-7f3e-4af1-bcbd-cff6abd178aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d7f566d-64nxd" Jul 27 21:04:06.741732 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:06.741708 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2307554c-7f3e-4af1-bcbd-cff6abd178aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d7f566d-64nxd\" (UID: \"2307554c-7f3e-4af1-bcbd-cff6abd178aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d7f566d-64nxd" Jul 27 21:04:06.806638 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:06.806581 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d7f566d-64nxd" Jul 27 21:04:06.964502 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:06.964468 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d7f566d-64nxd"] Jul 27 21:04:06.965831 ip-10-0-137-141 kubenswrapper[2570]: W0727 21:04:06.965806 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2307554c_7f3e_4af1_bcbd_cff6abd178aa.slice/crio-a47433d85426469af9364ed7896fa1897437c8d772da3458bf1a89c087c05016 WatchSource:0}: Error finding container a47433d85426469af9364ed7896fa1897437c8d772da3458bf1a89c087c05016: Status 404 returned error can't find the container with id a47433d85426469af9364ed7896fa1897437c8d772da3458bf1a89c087c05016 Jul 27 21:04:07.018018 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:07.017986 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d7f566d-64nxd" event={"ID":"2307554c-7f3e-4af1-bcbd-cff6abd178aa","Type":"ContainerStarted","Data":"a47433d85426469af9364ed7896fa1897437c8d772da3458bf1a89c087c05016"} Jul 27 21:04:08.026059 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:08.025968 2570 generic.go:356] "Generic (PLEG): container finished" podID="2307554c-7f3e-4af1-bcbd-cff6abd178aa" containerID="2f990baae4d3e63d2baa8c4788911d63bcd30612b67d7b538c966d2d6230cbe1" exitCode=0 Jul 27 21:04:08.026059 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:08.026019 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d7f566d-64nxd" event={"ID":"2307554c-7f3e-4af1-bcbd-cff6abd178aa","Type":"ContainerDied","Data":"2f990baae4d3e63d2baa8c4788911d63bcd30612b67d7b538c966d2d6230cbe1"} Jul 27 21:04:09.171373 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:09.171347 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d7f566d-64nxd" Jul 27 21:04:09.283294 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:09.283214 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2307554c-7f3e-4af1-bcbd-cff6abd178aa-maas-api-service-ca\") pod \"2307554c-7f3e-4af1-bcbd-cff6abd178aa\" (UID: \"2307554c-7f3e-4af1-bcbd-cff6abd178aa\") " Jul 27 21:04:09.283588 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:09.283565 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2307554c-7f3e-4af1-bcbd-cff6abd178aa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2307554c-7f3e-4af1-bcbd-cff6abd178aa" (UID: "2307554c-7f3e-4af1-bcbd-cff6abd178aa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:04:09.384755 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:09.384713 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2307554c-7f3e-4af1-bcbd-cff6abd178aa-maas-api-service-ca\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 21:04:10.039385 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:10.039357 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d7f566d-64nxd" Jul 27 21:04:10.039570 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:10.039399 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d7f566d-64nxd" event={"ID":"2307554c-7f3e-4af1-bcbd-cff6abd178aa","Type":"ContainerDied","Data":"a47433d85426469af9364ed7896fa1897437c8d772da3458bf1a89c087c05016"} Jul 27 21:04:10.039570 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:10.039449 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a47433d85426469af9364ed7896fa1897437c8d772da3458bf1a89c087c05016" Jul 27 21:04:16.907452 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:16.907409 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn"] Jul 27 21:04:16.908237 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:16.908205 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn" podUID="be0575a0-04d2-41c2-8f86-e8b6f1f2e112" containerName="maas-api" containerID="cri-o://f55375a03e0a04c2601a14efbcea16c1de12f2902e782d4166b85444a88eb629" gracePeriod=30 Jul 27 21:04:16.968395 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:16.968342 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4d7f566d-7c8df99c5c-6v4t6"] Jul 27 21:04:16.968699 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:16.968642 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-4d7f566d-7c8df99c5c-6v4t6" podUID="07615656-d386-477b-81c8-75786ff768c6" containerName="payload-processing" containerID="cri-o://2c5855e0a3f3b47f10319935bc6de9e8189bfa7860fe14923cfd479df7903c74" gracePeriod=30 Jul 27 21:04:17.089037 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:17.088993 2570 generic.go:356] "Generic (PLEG): container finished" podID="07615656-d386-477b-81c8-75786ff768c6" containerID="2c5855e0a3f3b47f10319935bc6de9e8189bfa7860fe14923cfd479df7903c74" exitCode=0 Jul 27 21:04:17.089196 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:17.089095 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4d7f566d-7c8df99c5c-6v4t6" event={"ID":"07615656-d386-477b-81c8-75786ff768c6","Type":"ContainerDied","Data":"2c5855e0a3f3b47f10319935bc6de9e8189bfa7860fe14923cfd479df7903c74"} Jul 27 21:04:17.091290 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:17.091265 2570 generic.go:356] "Generic (PLEG): container finished" podID="be0575a0-04d2-41c2-8f86-e8b6f1f2e112" containerID="f55375a03e0a04c2601a14efbcea16c1de12f2902e782d4166b85444a88eb629" exitCode=0 Jul 27 21:04:17.091411 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:17.091347 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn" event={"ID":"be0575a0-04d2-41c2-8f86-e8b6f1f2e112","Type":"ContainerDied","Data":"f55375a03e0a04c2601a14efbcea16c1de12f2902e782d4166b85444a88eb629"} Jul 27 21:04:17.231587 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:17.231559 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn" Jul 27 21:04:17.257295 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:17.257263 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpbjm\" (UniqueName: \"kubernetes.io/projected/be0575a0-04d2-41c2-8f86-e8b6f1f2e112-kube-api-access-qpbjm\") pod \"be0575a0-04d2-41c2-8f86-e8b6f1f2e112\" (UID: \"be0575a0-04d2-41c2-8f86-e8b6f1f2e112\") " Jul 27 21:04:17.257422 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:17.257396 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be0575a0-04d2-41c2-8f86-e8b6f1f2e112-openshift-service-ca\") pod \"be0575a0-04d2-41c2-8f86-e8b6f1f2e112\" (UID: \"be0575a0-04d2-41c2-8f86-e8b6f1f2e112\") " Jul 27 21:04:17.257481 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:17.257432 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be0575a0-04d2-41c2-8f86-e8b6f1f2e112-maas-api-tls\") pod \"be0575a0-04d2-41c2-8f86-e8b6f1f2e112\" (UID: \"be0575a0-04d2-41c2-8f86-e8b6f1f2e112\") " Jul 27 21:04:17.257823 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:17.257796 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be0575a0-04d2-41c2-8f86-e8b6f1f2e112-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "be0575a0-04d2-41c2-8f86-e8b6f1f2e112" (UID: "be0575a0-04d2-41c2-8f86-e8b6f1f2e112"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:04:17.259763 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:17.259738 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be0575a0-04d2-41c2-8f86-e8b6f1f2e112-kube-api-access-qpbjm" (OuterVolumeSpecName: "kube-api-access-qpbjm") pod "be0575a0-04d2-41c2-8f86-e8b6f1f2e112" (UID: "be0575a0-04d2-41c2-8f86-e8b6f1f2e112"). InnerVolumeSpecName "kube-api-access-qpbjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:04:17.259763 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:17.259738 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be0575a0-04d2-41c2-8f86-e8b6f1f2e112-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "be0575a0-04d2-41c2-8f86-e8b6f1f2e112" (UID: "be0575a0-04d2-41c2-8f86-e8b6f1f2e112"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 21:04:17.266199 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:17.266175 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4d7f566d-7c8df99c5c-6v4t6" Jul 27 21:04:17.358090 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:17.358053 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8hcn\" (UniqueName: \"kubernetes.io/projected/07615656-d386-477b-81c8-75786ff768c6-kube-api-access-h8hcn\") pod \"07615656-d386-477b-81c8-75786ff768c6\" (UID: \"07615656-d386-477b-81c8-75786ff768c6\") " Jul 27 21:04:17.358090 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:17.358099 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07615656-d386-477b-81c8-75786ff768c6-plugins-config-volume\") pod \"07615656-d386-477b-81c8-75786ff768c6\" (UID: \"07615656-d386-477b-81c8-75786ff768c6\") " Jul 27 21:04:17.358367 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:17.358334 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be0575a0-04d2-41c2-8f86-e8b6f1f2e112-maas-api-tls\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 21:04:17.358367 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:17.358345 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpbjm\" (UniqueName: \"kubernetes.io/projected/be0575a0-04d2-41c2-8f86-e8b6f1f2e112-kube-api-access-qpbjm\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 21:04:17.358367 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:17.358355 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be0575a0-04d2-41c2-8f86-e8b6f1f2e112-openshift-service-ca\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 21:04:17.358519 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:17.358485 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07615656-d386-477b-81c8-75786ff768c6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "07615656-d386-477b-81c8-75786ff768c6" (UID: "07615656-d386-477b-81c8-75786ff768c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:04:17.360387 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:17.360362 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07615656-d386-477b-81c8-75786ff768c6-kube-api-access-h8hcn" (OuterVolumeSpecName: "kube-api-access-h8hcn") pod "07615656-d386-477b-81c8-75786ff768c6" (UID: "07615656-d386-477b-81c8-75786ff768c6"). InnerVolumeSpecName "kube-api-access-h8hcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:04:17.459347 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:17.459251 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8hcn\" (UniqueName: \"kubernetes.io/projected/07615656-d386-477b-81c8-75786ff768c6-kube-api-access-h8hcn\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 21:04:17.459347 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:17.459282 2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07615656-d386-477b-81c8-75786ff768c6-plugins-config-volume\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 21:04:18.104182 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:18.104152 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn" Jul 27 21:04:18.104182 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:18.104175 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn" event={"ID":"be0575a0-04d2-41c2-8f86-e8b6f1f2e112","Type":"ContainerDied","Data":"01842a605fe5b256a96c6d5cb5b1ee91de045ffae88d6e9181961b069b28c836"} Jul 27 21:04:18.104759 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:18.104248 2570 scope.go:117] "RemoveContainer" containerID="f55375a03e0a04c2601a14efbcea16c1de12f2902e782d4166b85444a88eb629" Jul 27 21:04:18.105641 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:18.105617 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4d7f566d-7c8df99c5c-6v4t6" Jul 27 21:04:18.105641 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:18.105623 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4d7f566d-7c8df99c5c-6v4t6" event={"ID":"07615656-d386-477b-81c8-75786ff768c6","Type":"ContainerDied","Data":"50b7b57534a92c7a756f2cf162b7819dafbb2b7a89ccaf99fa71b960703543a3"} Jul 27 21:04:18.114593 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:18.114575 2570 scope.go:117] "RemoveContainer" containerID="2c5855e0a3f3b47f10319935bc6de9e8189bfa7860fe14923cfd479df7903c74" Jul 27 21:04:18.130784 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:18.130752 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4d7f566d-7c8df99c5c-6v4t6"] Jul 27 21:04:18.135691 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:18.135664 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4d7f566d-7c8df99c5c-6v4t6"] Jul 27 21:04:18.146402 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:18.146376 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn"] Jul 27 21:04:18.150142 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:18.150119 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4d7f566d-59f76f99cc-49pkn"] Jul 27 21:04:18.854724 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:18.854692 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07615656-d386-477b-81c8-75786ff768c6" path="/var/lib/kubelet/pods/07615656-d386-477b-81c8-75786ff768c6/volumes" Jul 27 21:04:18.855100 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:18.855087 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be0575a0-04d2-41c2-8f86-e8b6f1f2e112" path="/var/lib/kubelet/pods/be0575a0-04d2-41c2-8f86-e8b6f1f2e112/volumes" Jul 27 21:04:21.566970 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:21.566887 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d7f566d-64nxd"] Jul 27 21:04:21.571928 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:21.571898 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d7f566d-64nxd"] Jul 27 21:04:22.854839 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:22.854800 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2307554c-7f3e-4af1-bcbd-cff6abd178aa" path="/var/lib/kubelet/pods/2307554c-7f3e-4af1-bcbd-cff6abd178aa/volumes" Jul 27 21:04:24.751788 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:24.751749 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4"] Jul 27 21:04:24.752294 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:24.752278 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2307554c-7f3e-4af1-bcbd-cff6abd178aa" containerName="revoke-keys" Jul 27 21:04:24.752362 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:24.752298 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2307554c-7f3e-4af1-bcbd-cff6abd178aa" containerName="revoke-keys" Jul 27 21:04:24.752362 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:24.752331 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07615656-d386-477b-81c8-75786ff768c6" containerName="payload-processing" Jul 27 21:04:24.752362 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:24.752337 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="07615656-d386-477b-81c8-75786ff768c6" containerName="payload-processing" Jul 27 21:04:24.752362 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:24.752343 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be0575a0-04d2-41c2-8f86-e8b6f1f2e112" containerName="maas-api" Jul 27 21:04:24.752362 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:24.752350 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="be0575a0-04d2-41c2-8f86-e8b6f1f2e112" containerName="maas-api" Jul 27 21:04:24.752512 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:24.752423 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="07615656-d386-477b-81c8-75786ff768c6" containerName="payload-processing" Jul 27 21:04:24.752512 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:24.752432 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="be0575a0-04d2-41c2-8f86-e8b6f1f2e112" containerName="maas-api" Jul 27 21:04:24.752512 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:24.752441 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2307554c-7f3e-4af1-bcbd-cff6abd178aa" containerName="revoke-keys" Jul 27 21:04:24.755948 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:24.755926 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4" Jul 27 21:04:24.759578 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:24.759554 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-560b69a4-serving-cert\"" Jul 27 21:04:24.759799 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:24.759591 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cvmwf\"" Jul 27 21:04:24.772960 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:24.772924 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4"] Jul 27 21:04:24.826878 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:24.826832 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcp78\" (UniqueName: \"kubernetes.io/projected/2de34fca-ad60-470b-80e7-083273a8a9ba-kube-api-access-dcp78\") pod \"maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4\" (UID: \"2de34fca-ad60-470b-80e7-083273a8a9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4" Jul 27 21:04:24.827070 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:24.826905 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2de34fca-ad60-470b-80e7-083273a8a9ba-maas-api-tls\") pod \"maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4\" (UID: \"2de34fca-ad60-470b-80e7-083273a8a9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4" Jul 27 21:04:24.827070 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:24.827028 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2de34fca-ad60-470b-80e7-083273a8a9ba-openshift-service-ca\") pod \"maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4\" (UID: \"2de34fca-ad60-470b-80e7-083273a8a9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4" Jul 27 21:04:24.928301 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:24.928251 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2de34fca-ad60-470b-80e7-083273a8a9ba-openshift-service-ca\") pod \"maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4\" (UID: \"2de34fca-ad60-470b-80e7-083273a8a9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4" Jul 27 21:04:24.928521 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:24.928319 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcp78\" (UniqueName: \"kubernetes.io/projected/2de34fca-ad60-470b-80e7-083273a8a9ba-kube-api-access-dcp78\") pod \"maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4\" (UID: \"2de34fca-ad60-470b-80e7-083273a8a9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4" Jul 27 21:04:24.928521 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:24.928362 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2de34fca-ad60-470b-80e7-083273a8a9ba-maas-api-tls\") pod \"maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4\" (UID: \"2de34fca-ad60-470b-80e7-083273a8a9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4" Jul 27 21:04:24.928521 ip-10-0-137-141 kubenswrapper[2570]: E0727 21:04:24.928489 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-560b69a4-serving-cert: secret "maas-api-e2e-ait-560b69a4-serving-cert" not found Jul 27 21:04:24.928722 ip-10-0-137-141 kubenswrapper[2570]: E0727 21:04:24.928566 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2de34fca-ad60-470b-80e7-083273a8a9ba-maas-api-tls podName:2de34fca-ad60-470b-80e7-083273a8a9ba nodeName:}" failed. No retries permitted until 2026-07-27 21:04:25.428545042 +0000 UTC m=+1795.164525306 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2de34fca-ad60-470b-80e7-083273a8a9ba-maas-api-tls") pod "maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4" (UID: "2de34fca-ad60-470b-80e7-083273a8a9ba") : secret "maas-api-e2e-ait-560b69a4-serving-cert" not found Jul 27 21:04:24.929016 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:24.928995 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2de34fca-ad60-470b-80e7-083273a8a9ba-openshift-service-ca\") pod \"maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4\" (UID: \"2de34fca-ad60-470b-80e7-083273a8a9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4" Jul 27 21:04:24.938998 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:24.938956 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcp78\" (UniqueName: \"kubernetes.io/projected/2de34fca-ad60-470b-80e7-083273a8a9ba-kube-api-access-dcp78\") pod \"maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4\" (UID: \"2de34fca-ad60-470b-80e7-083273a8a9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4" Jul 27 21:04:25.432544 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:25.432506 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2de34fca-ad60-470b-80e7-083273a8a9ba-maas-api-tls\") pod \"maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4\" (UID: \"2de34fca-ad60-470b-80e7-083273a8a9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4" Jul 27 21:04:25.435202 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:25.435178 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2de34fca-ad60-470b-80e7-083273a8a9ba-maas-api-tls\") pod \"maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4\" (UID: \"2de34fca-ad60-470b-80e7-083273a8a9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4" Jul 27 21:04:25.672156 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:25.672123 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4" Jul 27 21:04:25.825925 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:25.825896 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4"] Jul 27 21:04:25.827776 ip-10-0-137-141 kubenswrapper[2570]: W0727 21:04:25.827734 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2de34fca_ad60_470b_80e7_083273a8a9ba.slice/crio-5391f79ecbd9db5145249e42de2e0fabd4fe5535a8d3e678fc7319953bba1ab5 WatchSource:0}: Error finding container 5391f79ecbd9db5145249e42de2e0fabd4fe5535a8d3e678fc7319953bba1ab5: Status 404 returned error can't find the container with id 5391f79ecbd9db5145249e42de2e0fabd4fe5535a8d3e678fc7319953bba1ab5 Jul 27 21:04:26.159895 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:26.159862 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4" event={"ID":"2de34fca-ad60-470b-80e7-083273a8a9ba","Type":"ContainerStarted","Data":"5391f79ecbd9db5145249e42de2e0fabd4fe5535a8d3e678fc7319953bba1ab5"} Jul 27 21:04:27.170162 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:27.170116 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4" event={"ID":"2de34fca-ad60-470b-80e7-083273a8a9ba","Type":"ContainerStarted","Data":"8b5a76e402c22b24aab73c2d64c33278793201ba97e7a19cade3341be4558e12"} Jul 27 21:04:27.170678 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:27.170241 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4" Jul 27 21:04:27.187724 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:27.187658 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4" podStartSLOduration=2.616963829 podStartE2EDuration="3.187642792s" totalImagesPullingTime="570.678963ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:27.187160016 +0000 UTC m=+1796.923140316" watchObservedRunningTime="2026-07-27 21:04:27.187642792 +0000 UTC m=+1796.923623080" Jul 27 21:04:33.185446 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:33.185417 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4" Jul 27 21:04:50.477131 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:50.477094 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-560b69a4-fwxwg"] Jul 27 21:04:50.481017 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:50.480996 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-560b69a4-fwxwg" Jul 27 21:04:50.483851 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:50.483826 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jzmp2\"" Jul 27 21:04:50.487589 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:50.487559 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-560b69a4-fwxwg"] Jul 27 21:04:50.579591 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:50.579551 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b6cb9a-6b91-4d16-81e1-e949bcf9f776-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-560b69a4-fwxwg\" (UID: \"f6b6cb9a-6b91-4d16-81e1-e949bcf9f776\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-560b69a4-fwxwg" Jul 27 21:04:50.681008 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:50.680969 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b6cb9a-6b91-4d16-81e1-e949bcf9f776-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-560b69a4-fwxwg\" (UID: \"f6b6cb9a-6b91-4d16-81e1-e949bcf9f776\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-560b69a4-fwxwg" Jul 27 21:04:50.681844 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:50.681813 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b6cb9a-6b91-4d16-81e1-e949bcf9f776-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-560b69a4-fwxwg\" (UID: \"f6b6cb9a-6b91-4d16-81e1-e949bcf9f776\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-560b69a4-fwxwg" Jul 27 21:04:50.797408 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:50.797318 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-560b69a4-fwxwg" Jul 27 21:04:50.942328 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:50.942294 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-560b69a4-fwxwg"] Jul 27 21:04:50.944160 ip-10-0-137-141 kubenswrapper[2570]: W0727 21:04:50.944137 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6b6cb9a_6b91_4d16_81e1_e949bcf9f776.slice/crio-0190ee2e1e6415db8d1927dd7f3881e9f0722fc5ce6724ca97e32cbaffb658c0 WatchSource:0}: Error finding container 0190ee2e1e6415db8d1927dd7f3881e9f0722fc5ce6724ca97e32cbaffb658c0: Status 404 returned error can't find the container with id 0190ee2e1e6415db8d1927dd7f3881e9f0722fc5ce6724ca97e32cbaffb658c0 Jul 27 21:04:51.330542 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:51.330507 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-560b69a4-fwxwg" event={"ID":"f6b6cb9a-6b91-4d16-81e1-e949bcf9f776","Type":"ContainerStarted","Data":"0190ee2e1e6415db8d1927dd7f3881e9f0722fc5ce6724ca97e32cbaffb658c0"} Jul 27 21:04:52.338494 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:52.338462 2570 generic.go:356] "Generic (PLEG): container finished" podID="f6b6cb9a-6b91-4d16-81e1-e949bcf9f776" containerID="631bb5988a8628ff85bf638f285d414560ecc7862918905aab0d1eb16d7491fc" exitCode=0 Jul 27 21:04:52.338929 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:52.338552 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-560b69a4-fwxwg" event={"ID":"f6b6cb9a-6b91-4d16-81e1-e949bcf9f776","Type":"ContainerDied","Data":"631bb5988a8628ff85bf638f285d414560ecc7862918905aab0d1eb16d7491fc"} Jul 27 21:04:53.494535 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:53.494502 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-560b69a4-fwxwg" Jul 27 21:04:53.611766 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:53.611678 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b6cb9a-6b91-4d16-81e1-e949bcf9f776-maas-api-service-ca\") pod \"f6b6cb9a-6b91-4d16-81e1-e949bcf9f776\" (UID: \"f6b6cb9a-6b91-4d16-81e1-e949bcf9f776\") " Jul 27 21:04:53.612069 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:53.612045 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6b6cb9a-6b91-4d16-81e1-e949bcf9f776-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f6b6cb9a-6b91-4d16-81e1-e949bcf9f776" (UID: "f6b6cb9a-6b91-4d16-81e1-e949bcf9f776"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:04:53.713385 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:53.713347 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b6cb9a-6b91-4d16-81e1-e949bcf9f776-maas-api-service-ca\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 21:04:54.350887 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:54.350853 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-560b69a4-fwxwg" event={"ID":"f6b6cb9a-6b91-4d16-81e1-e949bcf9f776","Type":"ContainerDied","Data":"0190ee2e1e6415db8d1927dd7f3881e9f0722fc5ce6724ca97e32cbaffb658c0"} Jul 27 21:04:54.351063 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:54.350893 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-560b69a4-fwxwg" Jul 27 21:04:54.351063 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:04:54.350901 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0190ee2e1e6415db8d1927dd7f3881e9f0722fc5ce6724ca97e32cbaffb658c0" Jul 27 21:05:01.052063 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:01.052020 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4"] Jul 27 21:05:01.052462 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:01.052240 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4" podUID="2de34fca-ad60-470b-80e7-083273a8a9ba" containerName="maas-api" containerID="cri-o://8b5a76e402c22b24aab73c2d64c33278793201ba97e7a19cade3341be4558e12" gracePeriod=30 Jul 27 21:05:01.326085 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:01.326057 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4" Jul 27 21:05:01.379233 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:01.379191 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcp78\" (UniqueName: \"kubernetes.io/projected/2de34fca-ad60-470b-80e7-083273a8a9ba-kube-api-access-dcp78\") pod \"2de34fca-ad60-470b-80e7-083273a8a9ba\" (UID: \"2de34fca-ad60-470b-80e7-083273a8a9ba\") " Jul 27 21:05:01.379423 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:01.379278 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2de34fca-ad60-470b-80e7-083273a8a9ba-openshift-service-ca\") pod \"2de34fca-ad60-470b-80e7-083273a8a9ba\" (UID: \"2de34fca-ad60-470b-80e7-083273a8a9ba\") " Jul 27 21:05:01.379423 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:01.379321 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2de34fca-ad60-470b-80e7-083273a8a9ba-maas-api-tls\") pod \"2de34fca-ad60-470b-80e7-083273a8a9ba\" (UID: \"2de34fca-ad60-470b-80e7-083273a8a9ba\") " Jul 27 21:05:01.379712 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:01.379685 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2de34fca-ad60-470b-80e7-083273a8a9ba-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2de34fca-ad60-470b-80e7-083273a8a9ba" (UID: "2de34fca-ad60-470b-80e7-083273a8a9ba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:05:01.381478 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:01.381450 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2de34fca-ad60-470b-80e7-083273a8a9ba-kube-api-access-dcp78" (OuterVolumeSpecName: "kube-api-access-dcp78") pod "2de34fca-ad60-470b-80e7-083273a8a9ba" (UID: "2de34fca-ad60-470b-80e7-083273a8a9ba"). InnerVolumeSpecName "kube-api-access-dcp78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:05:01.381575 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:01.381529 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2de34fca-ad60-470b-80e7-083273a8a9ba-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2de34fca-ad60-470b-80e7-083273a8a9ba" (UID: "2de34fca-ad60-470b-80e7-083273a8a9ba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 21:05:01.402590 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:01.402558 2570 generic.go:356] "Generic (PLEG): container finished" podID="2de34fca-ad60-470b-80e7-083273a8a9ba" containerID="8b5a76e402c22b24aab73c2d64c33278793201ba97e7a19cade3341be4558e12" exitCode=0 Jul 27 21:05:01.402762 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:01.402641 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4" event={"ID":"2de34fca-ad60-470b-80e7-083273a8a9ba","Type":"ContainerDied","Data":"8b5a76e402c22b24aab73c2d64c33278793201ba97e7a19cade3341be4558e12"} Jul 27 21:05:01.402762 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:01.402660 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4" Jul 27 21:05:01.402762 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:01.402739 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4" event={"ID":"2de34fca-ad60-470b-80e7-083273a8a9ba","Type":"ContainerDied","Data":"5391f79ecbd9db5145249e42de2e0fabd4fe5535a8d3e678fc7319953bba1ab5"} Jul 27 21:05:01.402932 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:01.402770 2570 scope.go:117] "RemoveContainer" containerID="8b5a76e402c22b24aab73c2d64c33278793201ba97e7a19cade3341be4558e12" Jul 27 21:05:01.415229 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:01.415209 2570 scope.go:117] "RemoveContainer" containerID="8b5a76e402c22b24aab73c2d64c33278793201ba97e7a19cade3341be4558e12" Jul 27 21:05:01.415702 ip-10-0-137-141 kubenswrapper[2570]: E0727 21:05:01.415676 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b5a76e402c22b24aab73c2d64c33278793201ba97e7a19cade3341be4558e12\": container with ID starting with 8b5a76e402c22b24aab73c2d64c33278793201ba97e7a19cade3341be4558e12 not found: ID does not exist" containerID="8b5a76e402c22b24aab73c2d64c33278793201ba97e7a19cade3341be4558e12" Jul 27 21:05:01.415783 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:01.415715 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b5a76e402c22b24aab73c2d64c33278793201ba97e7a19cade3341be4558e12"} err="failed to get container status \"8b5a76e402c22b24aab73c2d64c33278793201ba97e7a19cade3341be4558e12\": rpc error: code = NotFound desc = could not find container \"8b5a76e402c22b24aab73c2d64c33278793201ba97e7a19cade3341be4558e12\": container with ID starting with 8b5a76e402c22b24aab73c2d64c33278793201ba97e7a19cade3341be4558e12 not found: ID does not exist" Jul 27 21:05:01.426163 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:01.426140 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4"] Jul 27 21:05:01.431092 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:01.431070 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-560b69a4-69b9bb967f-9ssj4"] Jul 27 21:05:01.480847 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:01.480807 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2de34fca-ad60-470b-80e7-083273a8a9ba-maas-api-tls\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 21:05:01.481008 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:01.480852 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcp78\" (UniqueName: \"kubernetes.io/projected/2de34fca-ad60-470b-80e7-083273a8a9ba-kube-api-access-dcp78\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 21:05:01.481008 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:01.480868 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2de34fca-ad60-470b-80e7-083273a8a9ba-openshift-service-ca\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 21:05:02.853801 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:02.853768 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2de34fca-ad60-470b-80e7-083273a8a9ba" path="/var/lib/kubelet/pods/2de34fca-ad60-470b-80e7-083273a8a9ba/volumes" Jul 27 21:05:05.550632 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:05.550583 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-560b69a4-fwxwg"] Jul 27 21:05:05.553192 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:05.553163 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-560b69a4-fwxwg"] Jul 27 21:05:06.855018 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:06.854976 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6b6cb9a-6b91-4d16-81e1-e949bcf9f776" path="/var/lib/kubelet/pods/f6b6cb9a-6b91-4d16-81e1-e949bcf9f776/volumes" Jul 27 21:05:10.444338 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:10.444287 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj"] Jul 27 21:05:10.445036 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:10.445018 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6b6cb9a-6b91-4d16-81e1-e949bcf9f776" containerName="revoke-keys" Jul 27 21:05:10.445084 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:10.445039 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6b6cb9a-6b91-4d16-81e1-e949bcf9f776" containerName="revoke-keys" Jul 27 21:05:10.445084 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:10.445064 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2de34fca-ad60-470b-80e7-083273a8a9ba" containerName="maas-api" Jul 27 21:05:10.445084 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:10.445070 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2de34fca-ad60-470b-80e7-083273a8a9ba" containerName="maas-api" Jul 27 21:05:10.445195 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:10.445141 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2de34fca-ad60-470b-80e7-083273a8a9ba" containerName="maas-api" Jul 27 21:05:10.445195 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:10.445152 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6b6cb9a-6b91-4d16-81e1-e949bcf9f776" containerName="revoke-keys" Jul 27 21:05:10.449824 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:10.449796 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj" Jul 27 21:05:10.452623 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:10.452539 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-93e2aa82-serving-cert\"" Jul 27 21:05:10.453488 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:10.452783 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cvmwf\"" Jul 27 21:05:10.460186 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:10.460156 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj"] Jul 27 21:05:10.571819 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:10.571776 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506-maas-api-tls\") pod \"maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj\" (UID: \"ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj" Jul 27 21:05:10.572020 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:10.571834 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506-openshift-service-ca\") pod \"maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj\" (UID: \"ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj" Jul 27 21:05:10.572020 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:10.571930 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7c4g\" (UniqueName: \"kubernetes.io/projected/ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506-kube-api-access-f7c4g\") pod \"maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj\" (UID: \"ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj" Jul 27 21:05:10.673421 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:10.673382 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7c4g\" (UniqueName: \"kubernetes.io/projected/ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506-kube-api-access-f7c4g\") pod \"maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj\" (UID: \"ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj" Jul 27 21:05:10.673688 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:10.673472 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506-maas-api-tls\") pod \"maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj\" (UID: \"ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj" Jul 27 21:05:10.673688 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:10.673512 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506-openshift-service-ca\") pod \"maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj\" (UID: \"ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj" Jul 27 21:05:10.673688 ip-10-0-137-141 kubenswrapper[2570]: E0727 21:05:10.673644 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-93e2aa82-serving-cert: secret "maas-api-e2e-derive-93e2aa82-serving-cert" not found Jul 27 21:05:10.673862 ip-10-0-137-141 kubenswrapper[2570]: E0727 21:05:10.673709 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506-maas-api-tls podName:ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:11.173689946 +0000 UTC m=+1840.909670213 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506-maas-api-tls") pod "maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj" (UID: "ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506") : secret "maas-api-e2e-derive-93e2aa82-serving-cert" not found Jul 27 21:05:10.674259 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:10.674241 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506-openshift-service-ca\") pod \"maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj\" (UID: \"ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj" Jul 27 21:05:10.684450 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:10.684424 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7c4g\" (UniqueName: \"kubernetes.io/projected/ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506-kube-api-access-f7c4g\") pod \"maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj\" (UID: \"ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj" Jul 27 21:05:11.179274 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:11.179238 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506-maas-api-tls\") pod \"maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj\" (UID: \"ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj" Jul 27 21:05:11.182065 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:11.182026 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506-maas-api-tls\") pod \"maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj\" (UID: \"ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj" Jul 27 21:05:11.367457 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:11.367413 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj" Jul 27 21:05:11.532226 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:11.532193 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj"] Jul 27 21:05:11.533781 ip-10-0-137-141 kubenswrapper[2570]: W0727 21:05:11.533751 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee4a2cb9_ab6d_4ef5_86a6_f35f29b91506.slice/crio-feb69803100bc2ff434c144b754073a4033fdd474b6e0ee0f61e3bb9b1a84f79 WatchSource:0}: Error finding container feb69803100bc2ff434c144b754073a4033fdd474b6e0ee0f61e3bb9b1a84f79: Status 404 returned error can't find the container with id feb69803100bc2ff434c144b754073a4033fdd474b6e0ee0f61e3bb9b1a84f79 Jul 27 21:05:11.535142 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:11.535104 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 21:05:12.485656 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:12.485612 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj" event={"ID":"ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506","Type":"ContainerStarted","Data":"d0015334e16bbe213f53b48d3a5d02b9bf6ab4d0241a0156a6fa6d3f72c36d32"} Jul 27 21:05:12.485865 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:12.485672 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj" event={"ID":"ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506","Type":"ContainerStarted","Data":"feb69803100bc2ff434c144b754073a4033fdd474b6e0ee0f61e3bb9b1a84f79"} Jul 27 21:05:12.485865 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:12.485808 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj" Jul 27 21:05:12.506564 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:12.506506 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj" podStartSLOduration=2.104894929 podStartE2EDuration="2.50648919s" totalImagesPullingTime="401.594261ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:05:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:12.504184694 +0000 UTC m=+1842.240164992" watchObservedRunningTime="2026-07-27 21:05:12.50648919 +0000 UTC m=+1842.242469477" Jul 27 21:05:18.498377 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:18.498343 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj" Jul 27 21:05:19.351619 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:19.351565 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93e2aa82-6wlf7"] Jul 27 21:05:19.355711 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:19.355690 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93e2aa82-6wlf7" Jul 27 21:05:19.358213 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:19.358185 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jzmp2\"" Jul 27 21:05:19.362996 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:19.362968 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93e2aa82-6wlf7"] Jul 27 21:05:19.464327 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:19.464288 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb7d3aa3-6087-4b53-b21b-a41cf0ad71e1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-93e2aa82-6wlf7\" (UID: \"cb7d3aa3-6087-4b53-b21b-a41cf0ad71e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93e2aa82-6wlf7" Jul 27 21:05:19.565004 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:19.564961 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb7d3aa3-6087-4b53-b21b-a41cf0ad71e1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-93e2aa82-6wlf7\" (UID: \"cb7d3aa3-6087-4b53-b21b-a41cf0ad71e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93e2aa82-6wlf7" Jul 27 21:05:19.565670 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:19.565650 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb7d3aa3-6087-4b53-b21b-a41cf0ad71e1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-93e2aa82-6wlf7\" (UID: \"cb7d3aa3-6087-4b53-b21b-a41cf0ad71e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93e2aa82-6wlf7" Jul 27 21:05:19.668586 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:19.668478 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93e2aa82-6wlf7" Jul 27 21:05:19.817041 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:19.817014 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93e2aa82-6wlf7"] Jul 27 21:05:19.818630 ip-10-0-137-141 kubenswrapper[2570]: W0727 21:05:19.818575 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb7d3aa3_6087_4b53_b21b_a41cf0ad71e1.slice/crio-b02a01217cc04196ebe6c1b77a0676076377cf0c4b75c11f9a5b14e162c0175f WatchSource:0}: Error finding container b02a01217cc04196ebe6c1b77a0676076377cf0c4b75c11f9a5b14e162c0175f: Status 404 returned error can't find the container with id b02a01217cc04196ebe6c1b77a0676076377cf0c4b75c11f9a5b14e162c0175f Jul 27 21:05:20.538269 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:20.538226 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93e2aa82-6wlf7" event={"ID":"cb7d3aa3-6087-4b53-b21b-a41cf0ad71e1","Type":"ContainerStarted","Data":"a4f5e1f9ec4e00791204f3f5dccee7534c430560dd7f9370eb7cd466ffbd217b"} Jul 27 21:05:20.538472 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:20.538323 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93e2aa82-6wlf7" event={"ID":"cb7d3aa3-6087-4b53-b21b-a41cf0ad71e1","Type":"ContainerStarted","Data":"b02a01217cc04196ebe6c1b77a0676076377cf0c4b75c11f9a5b14e162c0175f"} Jul 27 21:05:20.559325 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:20.559262 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93e2aa82-6wlf7" podStartSLOduration=1.5592459509999999 podStartE2EDuration="1.559245951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:05:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:20.55583586 +0000 UTC m=+1850.291816147" watchObservedRunningTime="2026-07-27 21:05:20.559245951 +0000 UTC m=+1850.295226278" Jul 27 21:05:21.547216 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:21.547181 2570 generic.go:356] "Generic (PLEG): container finished" podID="cb7d3aa3-6087-4b53-b21b-a41cf0ad71e1" containerID="a4f5e1f9ec4e00791204f3f5dccee7534c430560dd7f9370eb7cd466ffbd217b" exitCode=0 Jul 27 21:05:21.547665 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:21.547284 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93e2aa82-6wlf7" event={"ID":"cb7d3aa3-6087-4b53-b21b-a41cf0ad71e1","Type":"ContainerDied","Data":"a4f5e1f9ec4e00791204f3f5dccee7534c430560dd7f9370eb7cd466ffbd217b"} Jul 27 21:05:22.701022 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:22.700985 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93e2aa82-6wlf7" Jul 27 21:05:22.795263 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:22.795224 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb7d3aa3-6087-4b53-b21b-a41cf0ad71e1-maas-api-service-ca\") pod \"cb7d3aa3-6087-4b53-b21b-a41cf0ad71e1\" (UID: \"cb7d3aa3-6087-4b53-b21b-a41cf0ad71e1\") " Jul 27 21:05:22.795627 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:22.795570 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb7d3aa3-6087-4b53-b21b-a41cf0ad71e1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cb7d3aa3-6087-4b53-b21b-a41cf0ad71e1" (UID: "cb7d3aa3-6087-4b53-b21b-a41cf0ad71e1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:05:22.896798 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:22.896763 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb7d3aa3-6087-4b53-b21b-a41cf0ad71e1-maas-api-service-ca\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 21:05:23.560956 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:23.560929 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93e2aa82-6wlf7" Jul 27 21:05:23.561168 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:23.560969 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93e2aa82-6wlf7" event={"ID":"cb7d3aa3-6087-4b53-b21b-a41cf0ad71e1","Type":"ContainerDied","Data":"b02a01217cc04196ebe6c1b77a0676076377cf0c4b75c11f9a5b14e162c0175f"} Jul 27 21:05:23.561168 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:23.561014 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b02a01217cc04196ebe6c1b77a0676076377cf0c4b75c11f9a5b14e162c0175f" Jul 27 21:05:30.190355 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:30.190312 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj"] Jul 27 21:05:30.190806 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:30.190614 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj" podUID="ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506" containerName="maas-api" containerID="cri-o://d0015334e16bbe213f53b48d3a5d02b9bf6ab4d0241a0156a6fa6d3f72c36d32" gracePeriod=30 Jul 27 21:05:30.471521 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:30.471481 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj" Jul 27 21:05:30.569089 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:30.569054 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506-maas-api-tls\") pod \"ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506\" (UID: \"ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506\") " Jul 27 21:05:30.569318 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:30.569276 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7c4g\" (UniqueName: \"kubernetes.io/projected/ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506-kube-api-access-f7c4g\") pod \"ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506\" (UID: \"ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506\") " Jul 27 21:05:30.569392 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:30.569318 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506-openshift-service-ca\") pod \"ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506\" (UID: \"ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506\") " Jul 27 21:05:30.569911 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:30.569878 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506" (UID: "ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:05:30.571511 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:30.571479 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506" (UID: "ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 21:05:30.572146 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:30.572112 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506-kube-api-access-f7c4g" (OuterVolumeSpecName: "kube-api-access-f7c4g") pod "ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506" (UID: "ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506"). InnerVolumeSpecName "kube-api-access-f7c4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:05:30.610533 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:30.610497 2570 generic.go:356] "Generic (PLEG): container finished" podID="ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506" containerID="d0015334e16bbe213f53b48d3a5d02b9bf6ab4d0241a0156a6fa6d3f72c36d32" exitCode=0 Jul 27 21:05:30.610731 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:30.610549 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj" Jul 27 21:05:30.610731 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:30.610594 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj" event={"ID":"ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506","Type":"ContainerDied","Data":"d0015334e16bbe213f53b48d3a5d02b9bf6ab4d0241a0156a6fa6d3f72c36d32"} Jul 27 21:05:30.610731 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:30.610668 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj" event={"ID":"ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506","Type":"ContainerDied","Data":"feb69803100bc2ff434c144b754073a4033fdd474b6e0ee0f61e3bb9b1a84f79"} Jul 27 21:05:30.610731 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:30.610690 2570 scope.go:117] "RemoveContainer" containerID="d0015334e16bbe213f53b48d3a5d02b9bf6ab4d0241a0156a6fa6d3f72c36d32" Jul 27 21:05:30.628651 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:30.628581 2570 scope.go:117] "RemoveContainer" containerID="d0015334e16bbe213f53b48d3a5d02b9bf6ab4d0241a0156a6fa6d3f72c36d32" Jul 27 21:05:30.628978 ip-10-0-137-141 kubenswrapper[2570]: E0727 21:05:30.628958 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0015334e16bbe213f53b48d3a5d02b9bf6ab4d0241a0156a6fa6d3f72c36d32\": container with ID starting with d0015334e16bbe213f53b48d3a5d02b9bf6ab4d0241a0156a6fa6d3f72c36d32 not found: ID does not exist" containerID="d0015334e16bbe213f53b48d3a5d02b9bf6ab4d0241a0156a6fa6d3f72c36d32" Jul 27 21:05:30.629023 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:30.628988 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0015334e16bbe213f53b48d3a5d02b9bf6ab4d0241a0156a6fa6d3f72c36d32"} err="failed to get container status \"d0015334e16bbe213f53b48d3a5d02b9bf6ab4d0241a0156a6fa6d3f72c36d32\": rpc error: code = NotFound desc = could not find container \"d0015334e16bbe213f53b48d3a5d02b9bf6ab4d0241a0156a6fa6d3f72c36d32\": container with ID starting with d0015334e16bbe213f53b48d3a5d02b9bf6ab4d0241a0156a6fa6d3f72c36d32 not found: ID does not exist" Jul 27 21:05:30.640647 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:30.640593 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj"] Jul 27 21:05:30.647314 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:30.647285 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-93e2aa82-69ffb759dd-b62qj"] Jul 27 21:05:30.671276 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:30.671235 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7c4g\" (UniqueName: \"kubernetes.io/projected/ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506-kube-api-access-f7c4g\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 21:05:30.671276 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:30.671275 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506-openshift-service-ca\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 21:05:30.671685 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:30.671287 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506-maas-api-tls\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 21:05:30.854943 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:30.854909 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506" path="/var/lib/kubelet/pods/ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506/volumes" Jul 27 21:05:34.447125 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:34.447086 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93e2aa82-6wlf7"] Jul 27 21:05:34.455009 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:34.454980 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93e2aa82-6wlf7"] Jul 27 21:05:34.854844 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:05:34.854800 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb7d3aa3-6087-4b53-b21b-a41cf0ad71e1" path="/var/lib/kubelet/pods/cb7d3aa3-6087-4b53-b21b-a41cf0ad71e1/volumes" Jul 27 21:07:16.060977 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:16.060940 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r"] Jul 27 21:07:16.061657 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:16.061641 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506" containerName="maas-api" Jul 27 21:07:16.061717 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:16.061660 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506" containerName="maas-api" Jul 27 21:07:16.061717 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:16.061685 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb7d3aa3-6087-4b53-b21b-a41cf0ad71e1" containerName="revoke-keys" Jul 27 21:07:16.061717 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:16.061694 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb7d3aa3-6087-4b53-b21b-a41cf0ad71e1" containerName="revoke-keys" Jul 27 21:07:16.061839 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:16.061796 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee4a2cb9-ab6d-4ef5-86a6-f35f29b91506" containerName="maas-api" Jul 27 21:07:16.061839 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:16.061815 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb7d3aa3-6087-4b53-b21b-a41cf0ad71e1" containerName="revoke-keys" Jul 27 21:07:16.065477 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:16.065459 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r" Jul 27 21:07:16.068160 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:16.068140 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2f806b4e-serving-cert\"" Jul 27 21:07:16.068753 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:16.068735 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cvmwf\"" Jul 27 21:07:16.076947 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:16.076922 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r"] Jul 27 21:07:16.202887 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:16.202837 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f4zt\" (UniqueName: \"kubernetes.io/projected/85431da1-3d85-4e98-98fb-bf7c3bc4de06-kube-api-access-5f4zt\") pod \"maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r\" (UID: \"85431da1-3d85-4e98-98fb-bf7c3bc4de06\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r" Jul 27 21:07:16.203086 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:16.202953 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85431da1-3d85-4e98-98fb-bf7c3bc4de06-openshift-service-ca\") pod \"maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r\" (UID: \"85431da1-3d85-4e98-98fb-bf7c3bc4de06\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r" Jul 27 21:07:16.203086 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:16.203015 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85431da1-3d85-4e98-98fb-bf7c3bc4de06-maas-api-tls\") pod \"maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r\" (UID: \"85431da1-3d85-4e98-98fb-bf7c3bc4de06\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r" Jul 27 21:07:16.304100 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:16.304059 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f4zt\" (UniqueName: \"kubernetes.io/projected/85431da1-3d85-4e98-98fb-bf7c3bc4de06-kube-api-access-5f4zt\") pod \"maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r\" (UID: \"85431da1-3d85-4e98-98fb-bf7c3bc4de06\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r" Jul 27 21:07:16.304303 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:16.304131 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85431da1-3d85-4e98-98fb-bf7c3bc4de06-openshift-service-ca\") pod \"maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r\" (UID: \"85431da1-3d85-4e98-98fb-bf7c3bc4de06\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r" Jul 27 21:07:16.304303 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:16.304167 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85431da1-3d85-4e98-98fb-bf7c3bc4de06-maas-api-tls\") pod \"maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r\" (UID: \"85431da1-3d85-4e98-98fb-bf7c3bc4de06\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r" Jul 27 21:07:16.304397 ip-10-0-137-141 kubenswrapper[2570]: E0727 21:07:16.304308 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-2f806b4e-serving-cert: secret "maas-api-e2e-mt-2f806b4e-serving-cert" not found Jul 27 21:07:16.304397 ip-10-0-137-141 kubenswrapper[2570]: E0727 21:07:16.304388 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85431da1-3d85-4e98-98fb-bf7c3bc4de06-maas-api-tls podName:85431da1-3d85-4e98-98fb-bf7c3bc4de06 nodeName:}" failed. No retries permitted until 2026-07-27 21:07:16.804372649 +0000 UTC m=+1966.540352913 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/85431da1-3d85-4e98-98fb-bf7c3bc4de06-maas-api-tls") pod "maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r" (UID: "85431da1-3d85-4e98-98fb-bf7c3bc4de06") : secret "maas-api-e2e-mt-2f806b4e-serving-cert" not found Jul 27 21:07:16.304996 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:16.304974 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85431da1-3d85-4e98-98fb-bf7c3bc4de06-openshift-service-ca\") pod \"maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r\" (UID: \"85431da1-3d85-4e98-98fb-bf7c3bc4de06\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r" Jul 27 21:07:16.315595 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:16.315520 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f4zt\" (UniqueName: \"kubernetes.io/projected/85431da1-3d85-4e98-98fb-bf7c3bc4de06-kube-api-access-5f4zt\") pod \"maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r\" (UID: \"85431da1-3d85-4e98-98fb-bf7c3bc4de06\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r" Jul 27 21:07:16.808859 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:16.808822 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85431da1-3d85-4e98-98fb-bf7c3bc4de06-maas-api-tls\") pod \"maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r\" (UID: \"85431da1-3d85-4e98-98fb-bf7c3bc4de06\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r" Jul 27 21:07:16.811631 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:16.811586 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85431da1-3d85-4e98-98fb-bf7c3bc4de06-maas-api-tls\") pod \"maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r\" (UID: \"85431da1-3d85-4e98-98fb-bf7c3bc4de06\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r" Jul 27 21:07:16.977244 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:16.977195 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r" Jul 27 21:07:17.132188 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:17.132159 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r"] Jul 27 21:07:17.133826 ip-10-0-137-141 kubenswrapper[2570]: W0727 21:07:17.133798 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85431da1_3d85_4e98_98fb_bf7c3bc4de06.slice/crio-b7d460edd51b954c057770c76ed07d235ddd6a1114673a426d0c0022027ef799 WatchSource:0}: Error finding container b7d460edd51b954c057770c76ed07d235ddd6a1114673a426d0c0022027ef799: Status 404 returned error can't find the container with id b7d460edd51b954c057770c76ed07d235ddd6a1114673a426d0c0022027ef799 Jul 27 21:07:17.284735 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:17.284695 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r" event={"ID":"85431da1-3d85-4e98-98fb-bf7c3bc4de06","Type":"ContainerStarted","Data":"b7d460edd51b954c057770c76ed07d235ddd6a1114673a426d0c0022027ef799"} Jul 27 21:07:18.294381 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:18.294343 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r" event={"ID":"85431da1-3d85-4e98-98fb-bf7c3bc4de06","Type":"ContainerStarted","Data":"ace0d7bb1253532d40ee4c6c308b1519cc0c7d79f78e71356f1ab301506786cc"} Jul 27 21:07:18.294819 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:18.294452 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r" Jul 27 21:07:18.312525 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:18.312462 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r" podStartSLOduration=1.8173010619999999 podStartE2EDuration="2.31244549s" totalImagesPullingTime="495.144428ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:07:18.311561223 +0000 UTC m=+1968.047541511" watchObservedRunningTime="2026-07-27 21:07:18.31244549 +0000 UTC m=+1968.048425778" Jul 27 21:07:24.307044 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:24.307003 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r" Jul 27 21:07:27.585733 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:27.585697 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2f806b4e-749kt"] Jul 27 21:07:27.589763 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:27.589744 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2f806b4e-749kt" Jul 27 21:07:27.592414 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:27.592385 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jzmp2\"" Jul 27 21:07:27.596668 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:27.596642 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2f806b4e-749kt"] Jul 27 21:07:27.715762 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:27.715724 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5086036-e28f-4a18-a6ce-44e528d3d9a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2f806b4e-749kt\" (UID: \"a5086036-e28f-4a18-a6ce-44e528d3d9a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2f806b4e-749kt" Jul 27 21:07:27.816770 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:27.816739 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5086036-e28f-4a18-a6ce-44e528d3d9a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2f806b4e-749kt\" (UID: \"a5086036-e28f-4a18-a6ce-44e528d3d9a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2f806b4e-749kt" Jul 27 21:07:27.817391 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:27.817372 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5086036-e28f-4a18-a6ce-44e528d3d9a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2f806b4e-749kt\" (UID: \"a5086036-e28f-4a18-a6ce-44e528d3d9a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2f806b4e-749kt" Jul 27 21:07:27.902307 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:27.902220 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2f806b4e-749kt" Jul 27 21:07:28.053441 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:28.053414 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2f806b4e-749kt"] Jul 27 21:07:28.055276 ip-10-0-137-141 kubenswrapper[2570]: W0727 21:07:28.055249 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5086036_e28f_4a18_a6ce_44e528d3d9a3.slice/crio-a80378dd2eb28c8b79800e4fa32929164cc4ca5dfeb3da43d9f6f1ce918731d6 WatchSource:0}: Error finding container a80378dd2eb28c8b79800e4fa32929164cc4ca5dfeb3da43d9f6f1ce918731d6: Status 404 returned error can't find the container with id a80378dd2eb28c8b79800e4fa32929164cc4ca5dfeb3da43d9f6f1ce918731d6 Jul 27 21:07:28.358587 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:28.358548 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2f806b4e-749kt" event={"ID":"a5086036-e28f-4a18-a6ce-44e528d3d9a3","Type":"ContainerStarted","Data":"a80378dd2eb28c8b79800e4fa32929164cc4ca5dfeb3da43d9f6f1ce918731d6"} Jul 27 21:07:29.366584 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:29.366543 2570 generic.go:356] "Generic (PLEG): container finished" podID="a5086036-e28f-4a18-a6ce-44e528d3d9a3" containerID="8707ebf105f0ae4cd364b7ad8d880ee736d4352f9241112ef8abbc51b7c4d431" exitCode=0 Jul 27 21:07:29.367018 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:29.366621 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2f806b4e-749kt" event={"ID":"a5086036-e28f-4a18-a6ce-44e528d3d9a3","Type":"ContainerDied","Data":"8707ebf105f0ae4cd364b7ad8d880ee736d4352f9241112ef8abbc51b7c4d431"} Jul 27 21:07:30.516091 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:30.516066 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2f806b4e-749kt" Jul 27 21:07:30.647971 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:30.647874 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5086036-e28f-4a18-a6ce-44e528d3d9a3-maas-api-service-ca\") pod \"a5086036-e28f-4a18-a6ce-44e528d3d9a3\" (UID: \"a5086036-e28f-4a18-a6ce-44e528d3d9a3\") " Jul 27 21:07:30.648233 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:30.648210 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5086036-e28f-4a18-a6ce-44e528d3d9a3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a5086036-e28f-4a18-a6ce-44e528d3d9a3" (UID: "a5086036-e28f-4a18-a6ce-44e528d3d9a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:07:30.749462 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:30.749416 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5086036-e28f-4a18-a6ce-44e528d3d9a3-maas-api-service-ca\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 21:07:31.380031 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:31.379997 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2f806b4e-749kt" Jul 27 21:07:31.380210 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:31.380035 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2f806b4e-749kt" event={"ID":"a5086036-e28f-4a18-a6ce-44e528d3d9a3","Type":"ContainerDied","Data":"a80378dd2eb28c8b79800e4fa32929164cc4ca5dfeb3da43d9f6f1ce918731d6"} Jul 27 21:07:31.380210 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:31.380084 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a80378dd2eb28c8b79800e4fa32929164cc4ca5dfeb3da43d9f6f1ce918731d6" Jul 27 21:07:37.751434 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:37.751392 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r"] Jul 27 21:07:37.751962 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:37.751724 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r" podUID="85431da1-3d85-4e98-98fb-bf7c3bc4de06" containerName="maas-api" containerID="cri-o://ace0d7bb1253532d40ee4c6c308b1519cc0c7d79f78e71356f1ab301506786cc" gracePeriod=30 Jul 27 21:07:38.025238 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:38.025203 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r" Jul 27 21:07:38.221843 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:38.221803 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85431da1-3d85-4e98-98fb-bf7c3bc4de06-openshift-service-ca\") pod \"85431da1-3d85-4e98-98fb-bf7c3bc4de06\" (UID: \"85431da1-3d85-4e98-98fb-bf7c3bc4de06\") " Jul 27 21:07:38.222046 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:38.221922 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85431da1-3d85-4e98-98fb-bf7c3bc4de06-maas-api-tls\") pod \"85431da1-3d85-4e98-98fb-bf7c3bc4de06\" (UID: \"85431da1-3d85-4e98-98fb-bf7c3bc4de06\") " Jul 27 21:07:38.222046 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:38.221966 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5f4zt\" (UniqueName: \"kubernetes.io/projected/85431da1-3d85-4e98-98fb-bf7c3bc4de06-kube-api-access-5f4zt\") pod \"85431da1-3d85-4e98-98fb-bf7c3bc4de06\" (UID: \"85431da1-3d85-4e98-98fb-bf7c3bc4de06\") " Jul 27 21:07:38.222218 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:38.222191 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85431da1-3d85-4e98-98fb-bf7c3bc4de06-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "85431da1-3d85-4e98-98fb-bf7c3bc4de06" (UID: "85431da1-3d85-4e98-98fb-bf7c3bc4de06"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:07:38.224237 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:38.224204 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85431da1-3d85-4e98-98fb-bf7c3bc4de06-kube-api-access-5f4zt" (OuterVolumeSpecName: "kube-api-access-5f4zt") pod "85431da1-3d85-4e98-98fb-bf7c3bc4de06" (UID: "85431da1-3d85-4e98-98fb-bf7c3bc4de06"). InnerVolumeSpecName "kube-api-access-5f4zt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:07:38.224370 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:38.224274 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85431da1-3d85-4e98-98fb-bf7c3bc4de06-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "85431da1-3d85-4e98-98fb-bf7c3bc4de06" (UID: "85431da1-3d85-4e98-98fb-bf7c3bc4de06"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 21:07:38.323735 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:38.323638 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f4zt\" (UniqueName: \"kubernetes.io/projected/85431da1-3d85-4e98-98fb-bf7c3bc4de06-kube-api-access-5f4zt\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 21:07:38.323735 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:38.323672 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85431da1-3d85-4e98-98fb-bf7c3bc4de06-openshift-service-ca\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 21:07:38.323735 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:38.323685 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85431da1-3d85-4e98-98fb-bf7c3bc4de06-maas-api-tls\") on node \"ip-10-0-137-141.ec2.internal\" DevicePath \"\"" Jul 27 21:07:38.423007 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:38.422972 2570 generic.go:356] "Generic (PLEG): container finished" podID="85431da1-3d85-4e98-98fb-bf7c3bc4de06" containerID="ace0d7bb1253532d40ee4c6c308b1519cc0c7d79f78e71356f1ab301506786cc" exitCode=0 Jul 27 21:07:38.423179 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:38.423017 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r" Jul 27 21:07:38.423179 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:38.423060 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r" event={"ID":"85431da1-3d85-4e98-98fb-bf7c3bc4de06","Type":"ContainerDied","Data":"ace0d7bb1253532d40ee4c6c308b1519cc0c7d79f78e71356f1ab301506786cc"} Jul 27 21:07:38.423179 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:38.423112 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r" event={"ID":"85431da1-3d85-4e98-98fb-bf7c3bc4de06","Type":"ContainerDied","Data":"b7d460edd51b954c057770c76ed07d235ddd6a1114673a426d0c0022027ef799"} Jul 27 21:07:38.423179 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:38.423135 2570 scope.go:117] "RemoveContainer" containerID="ace0d7bb1253532d40ee4c6c308b1519cc0c7d79f78e71356f1ab301506786cc" Jul 27 21:07:38.439748 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:38.439726 2570 scope.go:117] "RemoveContainer" containerID="ace0d7bb1253532d40ee4c6c308b1519cc0c7d79f78e71356f1ab301506786cc" Jul 27 21:07:38.440080 ip-10-0-137-141 kubenswrapper[2570]: E0727 21:07:38.440053 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ace0d7bb1253532d40ee4c6c308b1519cc0c7d79f78e71356f1ab301506786cc\": container with ID starting with ace0d7bb1253532d40ee4c6c308b1519cc0c7d79f78e71356f1ab301506786cc not found: ID does not exist" containerID="ace0d7bb1253532d40ee4c6c308b1519cc0c7d79f78e71356f1ab301506786cc" Jul 27 21:07:38.440171 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:38.440092 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ace0d7bb1253532d40ee4c6c308b1519cc0c7d79f78e71356f1ab301506786cc"} err="failed to get container status \"ace0d7bb1253532d40ee4c6c308b1519cc0c7d79f78e71356f1ab301506786cc\": rpc error: code = NotFound desc = could not find container \"ace0d7bb1253532d40ee4c6c308b1519cc0c7d79f78e71356f1ab301506786cc\": container with ID starting with ace0d7bb1253532d40ee4c6c308b1519cc0c7d79f78e71356f1ab301506786cc not found: ID does not exist" Jul 27 21:07:38.454273 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:38.454241 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r"] Jul 27 21:07:38.461589 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:38.461554 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2f806b4e-7756c984b6-vsw4r"] Jul 27 21:07:38.854759 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:38.854715 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85431da1-3d85-4e98-98fb-bf7c3bc4de06" path="/var/lib/kubelet/pods/85431da1-3d85-4e98-98fb-bf7c3bc4de06/volumes" Jul 27 21:07:42.661235 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:42.661190 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2f806b4e-749kt"] Jul 27 21:07:42.664010 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:42.663982 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2f806b4e-749kt"] Jul 27 21:07:42.854462 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:07:42.854429 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5086036-e28f-4a18-a6ce-44e528d3d9a3" path="/var/lib/kubelet/pods/a5086036-e28f-4a18-a6ce-44e528d3d9a3/volumes" Jul 27 21:08:08.047113 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:08.047070 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-46drx_92cbe4b8-7401-447f-a231-e7da3f656e94/manager/0.log" Jul 27 21:08:08.171281 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:08.171248 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-xv5xd_0a117ca9-bd46-41c8-9b0c-5791ca8cb369/manager/0.log" Jul 27 21:08:08.306054 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:08.305947 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-84d5c7b66c-6ggpf_c61fd4ad-ed54-40b1-ba3f-13d2ded266f8/manager/0.log" Jul 27 21:08:08.448650 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:08.448621 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-mssp2_dfc930c7-c597-440b-a01e-b07a05ed50ff/server/0.log" Jul 27 21:08:08.572391 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:08.572307 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-5vxxm_a82dfdde-1dc6-4286-9e7d-d8cc6c18c2fc/manager/0.log" Jul 27 21:08:08.709067 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:08.709036 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5767d7b464-bmvmq_181cb227-a222-41b5-b8dd-f6270ba914eb/copy-manifests/0.log" Jul 27 21:08:08.728880 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:08.728845 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5767d7b464-bmvmq_181cb227-a222-41b5-b8dd-f6270ba914eb/manager/0.log" Jul 27 21:08:09.335017 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:09.334983 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-554975b44f-2rt5h_7d0c452f-3cf8-41c4-8018-8c9486409ad9/maas-api/0.log" Jul 27 21:08:09.464132 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:09.464100 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753100-bsfpt_ab72247f-df8e-435e-bdee-941ae990cceb/cleanup/0.log" Jul 27 21:08:09.575177 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:09.575136 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-547x8_50f446ce-7287-48c1-a674-ee3bcf97429e/postgres/0.log" Jul 27 21:08:10.345447 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:10.345414 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bzw4l_ddf99cb9-8be9-4c04-ae1d-aee3ba69e320/util/0.log" Jul 27 21:08:10.352617 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:10.352570 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bzw4l_ddf99cb9-8be9-4c04-ae1d-aee3ba69e320/pull/0.log" Jul 27 21:08:10.358272 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:10.358249 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bzw4l_ddf99cb9-8be9-4c04-ae1d-aee3ba69e320/extract/0.log" Jul 27 21:08:10.467052 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:10.467012 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzk52_9a32e2ac-8d7a-43de-afc2-3ebb48f3841a/pull/0.log" Jul 27 21:08:10.474959 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:10.474932 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzk52_9a32e2ac-8d7a-43de-afc2-3ebb48f3841a/extract/0.log" Jul 27 21:08:10.481031 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:10.481007 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzk52_9a32e2ac-8d7a-43de-afc2-3ebb48f3841a/util/0.log" Jul 27 21:08:10.596629 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:10.596532 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pqhrm_d9231931-f505-4891-a372-c0581651da0b/util/0.log" Jul 27 21:08:10.603515 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:10.603490 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pqhrm_d9231931-f505-4891-a372-c0581651da0b/pull/0.log" Jul 27 21:08:10.610391 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:10.610364 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pqhrm_d9231931-f505-4891-a372-c0581651da0b/extract/0.log" Jul 27 21:08:10.720405 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:10.720368 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t79wc_3d05b9d7-e5bc-44c0-8e0d-67c1181d9340/util/0.log" Jul 27 21:08:10.727372 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:10.727342 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t79wc_3d05b9d7-e5bc-44c0-8e0d-67c1181d9340/pull/0.log" Jul 27 21:08:10.733455 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:10.733432 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t79wc_3d05b9d7-e5bc-44c0-8e0d-67c1181d9340/extract/0.log" Jul 27 21:08:10.963411 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:10.963382 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-5xm4m_8dc05f60-181a-4b57-8e9f-547c6b773d40/manager/0.log" Jul 27 21:08:11.067909 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:11.067873 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-znhw2_b6f5fe4c-b4c4-4deb-a0ee-c7d7a9ad38f2/manager/0.log" Jul 27 21:08:11.297700 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:11.297585 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-bkjqh_7060638d-30df-4674-9973-1d9840f1da32/registry-server/0.log" Jul 27 21:08:11.427977 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:11.427938 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-5hkmk_253c0a6c-5f53-42e7-a2e7-c23b3c8c5702/manager/0.log" Jul 27 21:08:11.671879 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:11.671774 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-8k2jd_500a9e69-82e8-4e6e-bbb8-fc6631c306e6/manager/0.log" Jul 27 21:08:12.280196 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:12.280160 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57b999c94c-rx7lh_f242764c-560b-4ab8-a4c3-23d74e64a6d2/kube-auth-proxy/0.log" Jul 27 21:08:12.522509 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:12.522473 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-r7skk_83ed38b7-1ad3-4784-8165-65eb3e4e74a1/istio-proxy/1.log" Jul 27 21:08:12.771312 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:12.771272 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-87qg9_ceb7cfd9-2198-4f37-8e01-d5f1782c20f2/payload-processing/0.log" Jul 27 21:08:13.250306 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:13.250263 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-96t9v_05e8790d-dbde-4fe7-a9c8-95b7007f1467/main/0.log" Jul 27 21:08:13.369123 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:13.369096 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-zwt4w_76e6f21d-36e6-4ee0-ab0c-48669bb6e9d0/main/0.log" Jul 27 21:08:21.808659 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:21.808626 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zrhx5_c3561079-4fcf-46be-8d0d-9a20c39be6d9/global-pull-secret-syncer/0.log" Jul 27 21:08:21.893437 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:21.893397 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-l6lsr_c33a8970-0e52-4e5e-82bf-e8823e1f88d9/konnectivity-agent/0.log" Jul 27 21:08:21.978693 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:21.978622 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-141.ec2.internal_27125cf5e1c5c4dd4172f89e5d84d2fd/haproxy/0.log" Jul 27 21:08:26.594947 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:26.594916 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bzw4l_ddf99cb9-8be9-4c04-ae1d-aee3ba69e320/extract/0.log" Jul 27 21:08:26.621502 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:26.621470 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bzw4l_ddf99cb9-8be9-4c04-ae1d-aee3ba69e320/util/0.log" Jul 27 21:08:26.644462 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:26.644428 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bzw4l_ddf99cb9-8be9-4c04-ae1d-aee3ba69e320/pull/0.log" Jul 27 21:08:26.686501 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:26.686470 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzk52_9a32e2ac-8d7a-43de-afc2-3ebb48f3841a/extract/0.log" Jul 27 21:08:26.714826 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:26.714795 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzk52_9a32e2ac-8d7a-43de-afc2-3ebb48f3841a/util/0.log" Jul 27 21:08:26.738361 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:26.738326 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzk52_9a32e2ac-8d7a-43de-afc2-3ebb48f3841a/pull/0.log" Jul 27 21:08:26.764113 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:26.764078 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pqhrm_d9231931-f505-4891-a372-c0581651da0b/extract/0.log" Jul 27 21:08:26.790412 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:26.790382 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pqhrm_d9231931-f505-4891-a372-c0581651da0b/util/0.log" Jul 27 21:08:26.811481 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:26.811449 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pqhrm_d9231931-f505-4891-a372-c0581651da0b/pull/0.log" Jul 27 21:08:26.840223 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:26.840186 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t79wc_3d05b9d7-e5bc-44c0-8e0d-67c1181d9340/extract/0.log" Jul 27 21:08:26.861785 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:26.861669 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t79wc_3d05b9d7-e5bc-44c0-8e0d-67c1181d9340/util/0.log" Jul 27 21:08:26.886239 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:26.886207 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1t79wc_3d05b9d7-e5bc-44c0-8e0d-67c1181d9340/pull/0.log" Jul 27 21:08:27.306395 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:27.306350 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-5xm4m_8dc05f60-181a-4b57-8e9f-547c6b773d40/manager/0.log" Jul 27 21:08:27.332279 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:27.332243 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-znhw2_b6f5fe4c-b4c4-4deb-a0ee-c7d7a9ad38f2/manager/0.log" Jul 27 21:08:27.411608 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:27.411557 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-bkjqh_7060638d-30df-4674-9973-1d9840f1da32/registry-server/0.log" Jul 27 21:08:27.533185 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:27.533147 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-5hkmk_253c0a6c-5f53-42e7-a2e7-c23b3c8c5702/manager/0.log" Jul 27 21:08:27.673411 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:27.673306 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-8k2jd_500a9e69-82e8-4e6e-bbb8-fc6631c306e6/manager/0.log" Jul 27 21:08:29.389337 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:29.389308 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5f3715d7-d32e-4c67-b57a-a91e4df5fa22/alertmanager/0.log" Jul 27 21:08:29.415178 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:29.415148 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5f3715d7-d32e-4c67-b57a-a91e4df5fa22/config-reloader/0.log" Jul 27 21:08:29.441775 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:29.441745 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5f3715d7-d32e-4c67-b57a-a91e4df5fa22/kube-rbac-proxy-web/0.log" Jul 27 21:08:29.462489 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:29.462461 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5f3715d7-d32e-4c67-b57a-a91e4df5fa22/kube-rbac-proxy/0.log" Jul 27 21:08:29.482843 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:29.482812 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5f3715d7-d32e-4c67-b57a-a91e4df5fa22/kube-rbac-proxy-metric/0.log" Jul 27 21:08:29.502380 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:29.502352 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5f3715d7-d32e-4c67-b57a-a91e4df5fa22/prom-label-proxy/0.log" Jul 27 21:08:29.529172 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:29.529139 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5f3715d7-d32e-4c67-b57a-a91e4df5fa22/init-config-reloader/0.log" Jul 27 21:08:29.573590 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:29.573552 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-6s67c_151eada8-db02-4937-a9fd-25fbc88c8f20/cluster-monitoring-operator/0.log" Jul 27 21:08:29.603713 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:29.603678 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-qn6t9_212e1f86-6abe-442e-a852-3885e1db697e/kube-state-metrics/0.log" Jul 27 21:08:29.628154 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:29.628123 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-qn6t9_212e1f86-6abe-442e-a852-3885e1db697e/kube-rbac-proxy-main/0.log" Jul 27 21:08:29.653949 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:29.653922 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-qn6t9_212e1f86-6abe-442e-a852-3885e1db697e/kube-rbac-proxy-self/0.log" Jul 27 21:08:29.717688 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:29.717555 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-k2f67_4e2dc09e-afb4-433d-83ca-a2add0b7cc2e/monitoring-plugin/0.log" Jul 27 21:08:29.836407 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:29.836367 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mk7dq_f571cef4-c931-4351-b2af-11d41a5d5d6f/node-exporter/0.log" Jul 27 21:08:29.898773 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:29.898746 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mk7dq_f571cef4-c931-4351-b2af-11d41a5d5d6f/kube-rbac-proxy/0.log" Jul 27 21:08:29.951733 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:29.951703 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mk7dq_f571cef4-c931-4351-b2af-11d41a5d5d6f/init-textfile/0.log" Jul 27 21:08:30.328422 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:30.328378 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-6b59f_c79f7f0b-c4e0-424f-9067-10f0ffe0963b/prometheus-operator/0.log" Jul 27 21:08:30.348549 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:30.348511 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-6b59f_c79f7f0b-c4e0-424f-9067-10f0ffe0963b/kube-rbac-proxy/0.log" Jul 27 21:08:30.411247 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:30.411217 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-cb749fc9-pgnpl_5374772e-877c-4718-8ae1-7e397424d3bc/telemeter-client/0.log" Jul 27 21:08:30.433017 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:30.432990 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-cb749fc9-pgnpl_5374772e-877c-4718-8ae1-7e397424d3bc/reload/0.log" Jul 27 21:08:30.454153 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:30.454123 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-cb749fc9-pgnpl_5374772e-877c-4718-8ae1-7e397424d3bc/kube-rbac-proxy/0.log" Jul 27 21:08:32.036345 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:32.036305 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mmb4n/perf-node-gather-daemonset-clvsm"] Jul 27 21:08:32.036796 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:32.036788 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85431da1-3d85-4e98-98fb-bf7c3bc4de06" containerName="maas-api" Jul 27 21:08:32.036848 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:32.036800 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="85431da1-3d85-4e98-98fb-bf7c3bc4de06" containerName="maas-api" Jul 27 21:08:32.036848 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:32.036813 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5086036-e28f-4a18-a6ce-44e528d3d9a3" containerName="revoke-keys" Jul 27 21:08:32.036848 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:32.036819 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5086036-e28f-4a18-a6ce-44e528d3d9a3" containerName="revoke-keys" Jul 27 21:08:32.036961 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:32.036900 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="85431da1-3d85-4e98-98fb-bf7c3bc4de06" containerName="maas-api" Jul 27 21:08:32.036961 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:32.036911 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5086036-e28f-4a18-a6ce-44e528d3d9a3" containerName="revoke-keys" Jul 27 21:08:32.039930 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:32.039911 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mmb4n/perf-node-gather-daemonset-clvsm" Jul 27 21:08:32.042247 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:32.042225 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mmb4n\"/\"kube-root-ca.crt\"" Jul 27 21:08:32.043213 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:32.043192 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mmb4n\"/\"default-dockercfg-xnqkl\"" Jul 27 21:08:32.043347 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:32.043198 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mmb4n\"/\"openshift-service-ca.crt\"" Jul 27 21:08:32.049900 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:32.049878 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mmb4n/perf-node-gather-daemonset-clvsm"] Jul 27 21:08:32.127668 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:32.127635 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7d33e7ae-fd32-41e3-9297-1ae86fe03e56-lib-modules\") pod \"perf-node-gather-daemonset-clvsm\" (UID: \"7d33e7ae-fd32-41e3-9297-1ae86fe03e56\") " pod="openshift-must-gather-mmb4n/perf-node-gather-daemonset-clvsm" Jul 27 21:08:32.127823 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:32.127678 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mrks\" (UniqueName: \"kubernetes.io/projected/7d33e7ae-fd32-41e3-9297-1ae86fe03e56-kube-api-access-2mrks\") pod \"perf-node-gather-daemonset-clvsm\" (UID: \"7d33e7ae-fd32-41e3-9297-1ae86fe03e56\") " pod="openshift-must-gather-mmb4n/perf-node-gather-daemonset-clvsm" Jul 27 21:08:32.127823 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:32.127706 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7d33e7ae-fd32-41e3-9297-1ae86fe03e56-proc\") pod \"perf-node-gather-daemonset-clvsm\" (UID: \"7d33e7ae-fd32-41e3-9297-1ae86fe03e56\") " pod="openshift-must-gather-mmb4n/perf-node-gather-daemonset-clvsm" Jul 27 21:08:32.127823 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:32.127772 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7d33e7ae-fd32-41e3-9297-1ae86fe03e56-podres\") pod \"perf-node-gather-daemonset-clvsm\" (UID: \"7d33e7ae-fd32-41e3-9297-1ae86fe03e56\") " pod="openshift-must-gather-mmb4n/perf-node-gather-daemonset-clvsm" Jul 27 21:08:32.127823 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:32.127793 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7d33e7ae-fd32-41e3-9297-1ae86fe03e56-sys\") pod \"perf-node-gather-daemonset-clvsm\" (UID: \"7d33e7ae-fd32-41e3-9297-1ae86fe03e56\") " pod="openshift-must-gather-mmb4n/perf-node-gather-daemonset-clvsm" Jul 27 21:08:32.228840 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:32.228807 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7d33e7ae-fd32-41e3-9297-1ae86fe03e56-lib-modules\") pod \"perf-node-gather-daemonset-clvsm\" (UID: \"7d33e7ae-fd32-41e3-9297-1ae86fe03e56\") " pod="openshift-must-gather-mmb4n/perf-node-gather-daemonset-clvsm" Jul 27 21:08:32.229035 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:32.228853 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mrks\" (UniqueName: \"kubernetes.io/projected/7d33e7ae-fd32-41e3-9297-1ae86fe03e56-kube-api-access-2mrks\") pod \"perf-node-gather-daemonset-clvsm\" (UID: \"7d33e7ae-fd32-41e3-9297-1ae86fe03e56\") " pod="openshift-must-gather-mmb4n/perf-node-gather-daemonset-clvsm" Jul 27 21:08:32.229035 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:32.228877 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7d33e7ae-fd32-41e3-9297-1ae86fe03e56-proc\") pod \"perf-node-gather-daemonset-clvsm\" (UID: \"7d33e7ae-fd32-41e3-9297-1ae86fe03e56\") " pod="openshift-must-gather-mmb4n/perf-node-gather-daemonset-clvsm" Jul 27 21:08:32.229035 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:32.228914 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7d33e7ae-fd32-41e3-9297-1ae86fe03e56-podres\") pod \"perf-node-gather-daemonset-clvsm\" (UID: \"7d33e7ae-fd32-41e3-9297-1ae86fe03e56\") " pod="openshift-must-gather-mmb4n/perf-node-gather-daemonset-clvsm" Jul 27 21:08:32.229035 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:32.228930 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7d33e7ae-fd32-41e3-9297-1ae86fe03e56-sys\") pod \"perf-node-gather-daemonset-clvsm\" (UID: \"7d33e7ae-fd32-41e3-9297-1ae86fe03e56\") " pod="openshift-must-gather-mmb4n/perf-node-gather-daemonset-clvsm" Jul 27 21:08:32.229035 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:32.229026 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7d33e7ae-fd32-41e3-9297-1ae86fe03e56-lib-modules\") pod \"perf-node-gather-daemonset-clvsm\" (UID: \"7d33e7ae-fd32-41e3-9297-1ae86fe03e56\") " pod="openshift-must-gather-mmb4n/perf-node-gather-daemonset-clvsm" Jul 27 21:08:32.229236 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:32.229060 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7d33e7ae-fd32-41e3-9297-1ae86fe03e56-podres\") pod \"perf-node-gather-daemonset-clvsm\" (UID: \"7d33e7ae-fd32-41e3-9297-1ae86fe03e56\") " pod="openshift-must-gather-mmb4n/perf-node-gather-daemonset-clvsm" Jul 27 21:08:32.229236 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:32.229026 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7d33e7ae-fd32-41e3-9297-1ae86fe03e56-proc\") pod \"perf-node-gather-daemonset-clvsm\" (UID: \"7d33e7ae-fd32-41e3-9297-1ae86fe03e56\") " pod="openshift-must-gather-mmb4n/perf-node-gather-daemonset-clvsm" Jul 27 21:08:32.229236 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:32.229059 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7d33e7ae-fd32-41e3-9297-1ae86fe03e56-sys\") pod \"perf-node-gather-daemonset-clvsm\" (UID: \"7d33e7ae-fd32-41e3-9297-1ae86fe03e56\") " pod="openshift-must-gather-mmb4n/perf-node-gather-daemonset-clvsm" Jul 27 21:08:32.237976 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:32.237944 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mrks\" (UniqueName: \"kubernetes.io/projected/7d33e7ae-fd32-41e3-9297-1ae86fe03e56-kube-api-access-2mrks\") pod \"perf-node-gather-daemonset-clvsm\" (UID: \"7d33e7ae-fd32-41e3-9297-1ae86fe03e56\") " pod="openshift-must-gather-mmb4n/perf-node-gather-daemonset-clvsm" Jul 27 21:08:32.353099 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:32.353012 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mmb4n/perf-node-gather-daemonset-clvsm" Jul 27 21:08:32.479529 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:32.479496 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-rfmfv_41483315-8990-4cb6-9312-1c14c9476301/console-operator/0.log" Jul 27 21:08:32.501065 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:32.501034 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mmb4n/perf-node-gather-daemonset-clvsm"] Jul 27 21:08:32.502511 ip-10-0-137-141 kubenswrapper[2570]: W0727 21:08:32.502481 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7d33e7ae_fd32_41e3_9297_1ae86fe03e56.slice/crio-91254fb030fe4e901d4f79dfc5bce134cfa768d67b7c34a61322f01334085611 WatchSource:0}: Error finding container 91254fb030fe4e901d4f79dfc5bce134cfa768d67b7c34a61322f01334085611: Status 404 returned error can't find the container with id 91254fb030fe4e901d4f79dfc5bce134cfa768d67b7c34a61322f01334085611 Jul 27 21:08:32.768008 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:32.767968 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mmb4n/perf-node-gather-daemonset-clvsm" event={"ID":"7d33e7ae-fd32-41e3-9297-1ae86fe03e56","Type":"ContainerStarted","Data":"13bfbc144104498e394bad95886d58cbaa307d846cbc5f5095d933702576bbc0"} Jul 27 21:08:32.768224 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:32.768031 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mmb4n/perf-node-gather-daemonset-clvsm" event={"ID":"7d33e7ae-fd32-41e3-9297-1ae86fe03e56","Type":"ContainerStarted","Data":"91254fb030fe4e901d4f79dfc5bce134cfa768d67b7c34a61322f01334085611"} Jul 27 21:08:32.768224 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:32.768071 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mmb4n/perf-node-gather-daemonset-clvsm" Jul 27 21:08:32.787349 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:32.787297 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mmb4n/perf-node-gather-daemonset-clvsm" podStartSLOduration=0.787280339 podStartE2EDuration="787.280339ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:08:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:08:32.784501875 +0000 UTC m=+2042.520482172" watchObservedRunningTime="2026-07-27 21:08:32.787280339 +0000 UTC m=+2042.523260625" Jul 27 21:08:32.992172 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:32.992143 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5c9bc67bcd-7vfj5_ed3a3437-420b-4fd4-bdc6-5fece371b252/console/0.log" Jul 27 21:08:33.614877 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:33.614842 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-mnsh8_3a68f100-f17d-4f48-b78e-27ff219c64ac/volume-data-source-validator/0.log" Jul 27 21:08:34.632716 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:34.632672 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6h8jp_4c366dc4-9298-4f87-bcc6-9a8794efc1f7/dns/0.log" Jul 27 21:08:34.653519 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:34.653482 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6h8jp_4c366dc4-9298-4f87-bcc6-9a8794efc1f7/kube-rbac-proxy/0.log" Jul 27 21:08:34.771661 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:34.771618 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2gppc_8c122d10-fe63-4972-87c0-4cf309037cc8/dns-node-resolver/0.log" Jul 27 21:08:35.417886 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:35.417852 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vf9q2_c5f8ec4d-adb1-4938-8887-6708463e4b02/node-ca/0.log" Jul 27 21:08:36.450895 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:36.450858 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57b999c94c-rx7lh_f242764c-560b-4ab8-a4c3-23d74e64a6d2/kube-auth-proxy/0.log" Jul 27 21:08:36.506549 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:36.506518 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-r7skk_83ed38b7-1ad3-4784-8165-65eb3e4e74a1/istio-proxy/0.log" Jul 27 21:08:36.555372 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:36.555333 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-r7skk_83ed38b7-1ad3-4784-8165-65eb3e4e74a1/istio-proxy/1.log" Jul 27 21:08:36.621077 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:36.621042 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-87qg9_ceb7cfd9-2198-4f37-8e01-d5f1782c20f2/payload-processing/0.log" Jul 27 21:08:37.332682 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:37.332646 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4xgk6_b2788954-9e92-4c96-b86a-f2efb00df57d/serve-healthcheck-canary/0.log" Jul 27 21:08:38.028266 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:38.028240 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s4xkz_a4512db5-a1ae-4484-be6e-cd5b7b3046ee/kube-rbac-proxy/0.log" Jul 27 21:08:38.050029 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:38.050002 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s4xkz_a4512db5-a1ae-4484-be6e-cd5b7b3046ee/exporter/0.log" Jul 27 21:08:38.070980 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:38.070948 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s4xkz_a4512db5-a1ae-4484-be6e-cd5b7b3046ee/extractor/0.log" Jul 27 21:08:38.785438 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:38.785411 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mmb4n/perf-node-gather-daemonset-clvsm" Jul 27 21:08:40.088533 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:40.088501 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-46drx_92cbe4b8-7401-447f-a231-e7da3f656e94/manager/0.log" Jul 27 21:08:40.307810 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:40.307776 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-xv5xd_0a117ca9-bd46-41c8-9b0c-5791ca8cb369/manager/0.log" Jul 27 21:08:40.462755 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:40.462710 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-84d5c7b66c-6ggpf_c61fd4ad-ed54-40b1-ba3f-13d2ded266f8/manager/0.log" Jul 27 21:08:40.495116 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:40.495085 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-mssp2_dfc930c7-c597-440b-a01e-b07a05ed50ff/server/0.log" Jul 27 21:08:40.603366 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:40.603325 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-5vxxm_a82dfdde-1dc6-4286-9e7d-d8cc6c18c2fc/manager/0.log" Jul 27 21:08:40.714585 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:40.714496 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5767d7b464-bmvmq_181cb227-a222-41b5-b8dd-f6270ba914eb/manager/0.log" Jul 27 21:08:40.743914 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:40.743885 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5767d7b464-bmvmq_181cb227-a222-41b5-b8dd-f6270ba914eb/copy-manifests/0.log" Jul 27 21:08:42.281328 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:42.281294 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-lbx57_30871d9e-eee0-47b9-aa1b-6963cbaa54ea/openshift-lws-operator/0.log" Jul 27 21:08:42.798061 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:42.798032 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57b999c94c-rx7lh_f242764c-560b-4ab8-a4c3-23d74e64a6d2/kube-auth-proxy/0.log" Jul 27 21:08:42.831418 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:42.831377 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-r7skk_83ed38b7-1ad3-4784-8165-65eb3e4e74a1/istio-proxy/0.log" Jul 27 21:08:42.883580 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:42.883549 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-r7skk_83ed38b7-1ad3-4784-8165-65eb3e4e74a1/istio-proxy/1.log" Jul 27 21:08:42.944813 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:42.944781 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-87qg9_ceb7cfd9-2198-4f37-8e01-d5f1782c20f2/payload-processing/0.log" Jul 27 21:08:44.241839 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:44.241786 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57b999c94c-rx7lh_f242764c-560b-4ab8-a4c3-23d74e64a6d2/kube-auth-proxy/0.log" Jul 27 21:08:44.281615 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:44.281538 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-r7skk_83ed38b7-1ad3-4784-8165-65eb3e4e74a1/istio-proxy/0.log" Jul 27 21:08:44.330377 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:44.330316 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-r7skk_83ed38b7-1ad3-4784-8165-65eb3e4e74a1/istio-proxy/1.log" Jul 27 21:08:44.384983 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:44.384952 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-87qg9_ceb7cfd9-2198-4f37-8e01-d5f1782c20f2/payload-processing/0.log" Jul 27 21:08:48.763352 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:48.763323 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-22mpg_f0a531c5-371d-4024-9a40-a542d5cfdeb4/kube-multus/0.log" Jul 27 21:08:48.790412 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:48.790387 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ljtzv_35f7c023-fb9f-40ea-9f5f-547b26383bb8/kube-multus-additional-cni-plugins/0.log" Jul 27 21:08:48.812109 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:48.812079 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ljtzv_35f7c023-fb9f-40ea-9f5f-547b26383bb8/egress-router-binary-copy/0.log" Jul 27 21:08:48.834853 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:48.834821 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ljtzv_35f7c023-fb9f-40ea-9f5f-547b26383bb8/cni-plugins/0.log" Jul 27 21:08:48.858977 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:48.858945 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ljtzv_35f7c023-fb9f-40ea-9f5f-547b26383bb8/bond-cni-plugin/0.log" Jul 27 21:08:48.881556 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:48.881528 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ljtzv_35f7c023-fb9f-40ea-9f5f-547b26383bb8/routeoverride-cni/0.log" Jul 27 21:08:48.904363 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:48.904338 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ljtzv_35f7c023-fb9f-40ea-9f5f-547b26383bb8/whereabouts-cni-bincopy/0.log" Jul 27 21:08:48.926302 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:48.926267 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ljtzv_35f7c023-fb9f-40ea-9f5f-547b26383bb8/whereabouts-cni/0.log" Jul 27 21:08:49.395475 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:49.395445 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tkd2d_05ef8a1e-7bd9-4436-a007-618d151dfd5a/network-metrics-daemon/0.log" Jul 27 21:08:49.412957 ip-10-0-137-141 kubenswrapper[2570]: I0727 21:08:49.412925 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tkd2d_05ef8a1e-7bd9-4436-a007-618d151dfd5a/kube-rbac-proxy/0.log"