Jul 08 18:45:37.148611 ip-10-0-130-48 systemd[1]: Starting Kubernetes Kubelet... Jul 08 18:45:37.521670 ip-10-0-130-48 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 18:45:37.521670 ip-10-0-130-48 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 08 18:45:37.521670 ip-10-0-130-48 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 18:45:37.521670 ip-10-0-130-48 kubenswrapper[2570]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 08 18:45:37.521670 ip-10-0-130-48 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 18:45:37.522616 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.522527 2570 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 08 18:45:37.529977 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.529956 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 18:45:37.529977 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.529972 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 18:45:37.529977 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.529977 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 18:45:37.529977 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.529981 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 18:45:37.529977 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.529985 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 18:45:37.530274 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.529991 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 18:45:37.530274 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.529995 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 18:45:37.530274 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.529998 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 18:45:37.530274 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530002 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 18:45:37.530274 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530006 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 18:45:37.530274 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530010 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 18:45:37.530274 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530018 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 18:45:37.530274 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530022 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 18:45:37.530274 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530025 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 18:45:37.530274 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530031 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 18:45:37.530274 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530035 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 18:45:37.530274 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530039 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 18:45:37.530274 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530042 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 18:45:37.530274 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530046 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 18:45:37.530274 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530050 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 18:45:37.530274 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530053 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 18:45:37.530274 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530057 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 18:45:37.530274 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530061 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 18:45:37.530274 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530065 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 18:45:37.530869 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530071 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 18:45:37.530869 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530078 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 18:45:37.530869 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530082 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 18:45:37.530869 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530086 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 18:45:37.530869 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530090 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 18:45:37.530869 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530094 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 18:45:37.530869 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530098 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 18:45:37.530869 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530102 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 18:45:37.530869 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530106 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 08 18:45:37.530869 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530110 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 18:45:37.530869 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530114 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 18:45:37.530869 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530118 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 18:45:37.530869 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530121 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 18:45:37.530869 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530126 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 18:45:37.530869 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530132 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 18:45:37.530869 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530136 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 18:45:37.530869 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530140 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 18:45:37.530869 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530145 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 18:45:37.530869 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530150 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 18:45:37.530869 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530155 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 18:45:37.531544 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530159 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 18:45:37.531544 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530163 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 18:45:37.531544 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530167 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 18:45:37.531544 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530172 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 18:45:37.531544 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530177 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 18:45:37.531544 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530181 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 18:45:37.531544 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530188 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 18:45:37.531544 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530195 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 18:45:37.531544 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530200 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 18:45:37.531544 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530205 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 18:45:37.531544 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530209 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 18:45:37.531544 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530214 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 18:45:37.531544 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530219 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 18:45:37.531544 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530223 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 18:45:37.531544 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530227 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 18:45:37.531544 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530231 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 18:45:37.531544 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530235 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 18:45:37.531544 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530240 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 18:45:37.531544 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530244 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 18:45:37.532298 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530249 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 18:45:37.532298 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530254 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 18:45:37.532298 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530258 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 18:45:37.532298 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530262 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 18:45:37.532298 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530266 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 18:45:37.532298 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530270 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 18:45:37.532298 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530274 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 18:45:37.532298 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530279 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 18:45:37.532298 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530283 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 18:45:37.532298 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530288 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 18:45:37.532298 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530292 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 18:45:37.532298 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530296 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 18:45:37.532298 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530303 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 18:45:37.532298 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530323 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 18:45:37.532298 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530327 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 18:45:37.532298 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530331 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 18:45:37.532298 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530336 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 18:45:37.532298 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530340 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 18:45:37.532298 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530344 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 18:45:37.532298 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530348 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 18:45:37.533140 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530353 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 18:45:37.533140 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530360 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 18:45:37.533140 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530365 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 18:45:37.533140 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530371 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 18:45:37.533140 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530375 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 18:45:37.533140 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530381 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 18:45:37.533140 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530385 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 18:45:37.533140 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530389 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 18:45:37.533140 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530394 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 18:45:37.533140 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530399 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 18:45:37.533140 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530403 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 18:45:37.533140 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530408 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 18:45:37.533140 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530413 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 18:45:37.533140 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530417 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 18:45:37.533140 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530422 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 18:45:37.533140 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.530427 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 18:45:37.533140 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532229 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 18:45:37.533140 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532241 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 18:45:37.533917 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532246 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 18:45:37.533917 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532252 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 18:45:37.533917 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532256 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 18:45:37.533917 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532260 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 18:45:37.533917 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532265 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 18:45:37.533917 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532269 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 18:45:37.533917 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532273 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 18:45:37.533917 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532282 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 18:45:37.533917 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532288 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 18:45:37.533917 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532293 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 18:45:37.533917 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532298 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 18:45:37.533917 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532302 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 18:45:37.533917 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532325 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 08 18:45:37.533917 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532330 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 18:45:37.533917 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532335 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 18:45:37.533917 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532341 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 18:45:37.533917 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532347 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 18:45:37.533917 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532352 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 18:45:37.533917 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532356 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 18:45:37.534753 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532361 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 18:45:37.534753 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532365 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 18:45:37.534753 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532370 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 18:45:37.534753 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532374 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 18:45:37.534753 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532379 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 18:45:37.534753 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532383 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 18:45:37.534753 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532388 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 18:45:37.534753 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532392 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 18:45:37.534753 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532397 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 18:45:37.534753 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532401 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 18:45:37.534753 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532405 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 18:45:37.534753 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532410 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 18:45:37.534753 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532417 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 18:45:37.534753 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532421 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 18:45:37.534753 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532426 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 18:45:37.534753 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532430 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 18:45:37.534753 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532435 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 18:45:37.534753 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532439 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 18:45:37.534753 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532444 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 18:45:37.534753 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532449 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 18:45:37.535627 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532453 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 18:45:37.535627 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532459 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 18:45:37.535627 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532463 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 18:45:37.535627 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532467 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 18:45:37.535627 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532472 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 18:45:37.535627 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532476 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 18:45:37.535627 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532480 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 18:45:37.535627 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532484 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 18:45:37.535627 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532489 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 18:45:37.535627 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532493 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 18:45:37.535627 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532497 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 18:45:37.535627 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532501 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 18:45:37.535627 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532506 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 18:45:37.535627 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532510 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 18:45:37.535627 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532514 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 18:45:37.535627 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532518 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 18:45:37.535627 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532522 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 18:45:37.535627 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532527 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 18:45:37.535627 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532531 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 18:45:37.535627 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532535 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 18:45:37.536108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532539 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 18:45:37.536108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532545 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 18:45:37.536108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532551 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 18:45:37.536108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532556 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 18:45:37.536108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532560 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 18:45:37.536108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532565 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 18:45:37.536108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532570 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 18:45:37.536108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532575 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 18:45:37.536108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532579 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 18:45:37.536108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532583 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 18:45:37.536108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532588 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 18:45:37.536108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532594 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 18:45:37.536108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532598 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 18:45:37.536108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532602 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 18:45:37.536108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532608 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 18:45:37.536108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532612 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 18:45:37.536108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532617 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 18:45:37.536108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532621 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 18:45:37.536108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532625 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 18:45:37.536108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532630 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 18:45:37.536644 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532634 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 18:45:37.536644 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532639 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 18:45:37.536644 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532643 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 18:45:37.536644 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532647 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 18:45:37.536644 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532652 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 18:45:37.536644 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532657 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 18:45:37.536644 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532661 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 18:45:37.536644 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532665 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 18:45:37.536644 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532669 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 18:45:37.536644 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532684 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 18:45:37.536644 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532689 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 18:45:37.536644 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532693 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 18:45:37.536644 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532697 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 18:45:37.536644 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532702 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 18:45:37.536644 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532706 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 18:45:37.536644 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532710 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 18:45:37.536644 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532715 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 18:45:37.536644 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.532719 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 18:45:37.536644 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533372 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 18:45:37.537108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533382 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 18:45:37.537108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533387 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 08 18:45:37.537108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533391 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 18:45:37.537108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533396 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 18:45:37.537108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533400 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 18:45:37.537108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533405 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 18:45:37.537108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533409 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 18:45:37.537108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533414 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 18:45:37.537108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533418 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 18:45:37.537108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533423 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 18:45:37.537108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533428 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 18:45:37.537108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533432 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 18:45:37.537108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533437 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 18:45:37.537108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533441 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 18:45:37.537108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533445 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 18:45:37.537108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533450 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 18:45:37.537108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533454 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 18:45:37.537108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533458 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 18:45:37.537108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533462 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 18:45:37.537108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533466 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 18:45:37.537604 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533470 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 18:45:37.537604 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533474 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 18:45:37.537604 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533478 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 18:45:37.537604 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533482 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 18:45:37.537604 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533486 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 18:45:37.537604 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533490 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 18:45:37.537604 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533497 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 18:45:37.537604 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533503 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 18:45:37.537604 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533507 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 18:45:37.537604 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533511 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 18:45:37.537604 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533516 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 18:45:37.537604 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533520 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 18:45:37.537604 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533525 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 18:45:37.537604 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533529 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 18:45:37.537604 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533533 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 18:45:37.537604 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533537 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 18:45:37.537604 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533544 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 18:45:37.537604 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533549 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 18:45:37.537604 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533554 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 18:45:37.538084 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533559 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 18:45:37.538084 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533564 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 18:45:37.538084 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533569 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 18:45:37.538084 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533574 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 18:45:37.538084 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533578 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 18:45:37.538084 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533582 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 18:45:37.538084 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533586 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 18:45:37.538084 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533591 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 18:45:37.538084 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533595 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 18:45:37.538084 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533602 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 18:45:37.538084 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533607 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 18:45:37.538084 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533612 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 18:45:37.538084 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533617 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 18:45:37.538084 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533622 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 18:45:37.538084 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533627 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 18:45:37.538084 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533632 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 18:45:37.538084 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533636 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 18:45:37.538084 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533641 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 18:45:37.538084 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533645 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 18:45:37.538084 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533650 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 18:45:37.538580 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533654 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 18:45:37.538580 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533658 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 18:45:37.538580 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533663 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 18:45:37.538580 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533667 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 18:45:37.538580 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533671 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 18:45:37.538580 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533675 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 18:45:37.538580 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533679 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 18:45:37.538580 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533685 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 18:45:37.538580 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533689 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 18:45:37.538580 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533693 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 18:45:37.538580 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533698 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 18:45:37.538580 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533702 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 18:45:37.538580 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533706 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 18:45:37.538580 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533711 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 18:45:37.538580 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533716 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 18:45:37.538580 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533721 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 18:45:37.538580 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533725 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 18:45:37.538580 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533729 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 18:45:37.538580 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533734 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 18:45:37.538580 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533738 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 18:45:37.539053 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533742 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 18:45:37.539053 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533746 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 18:45:37.539053 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533750 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 18:45:37.539053 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533754 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 18:45:37.539053 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533758 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 18:45:37.539053 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533763 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 18:45:37.539053 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533767 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 18:45:37.539053 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533772 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 18:45:37.539053 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533776 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 18:45:37.539053 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533780 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 18:45:37.539053 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533784 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 18:45:37.539053 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533789 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 18:45:37.539053 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533793 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 18:45:37.539053 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533797 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 18:45:37.539053 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533801 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 18:45:37.539053 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533805 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 18:45:37.539053 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533810 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 18:45:37.539053 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533817 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 18:45:37.539053 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.533821 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 18:45:37.539053 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.533931 2570 flags.go:64] FLAG: --address="0.0.0.0" Jul 08 18:45:37.539576 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.533943 2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 08 18:45:37.539576 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.533955 2570 flags.go:64] FLAG: --anonymous-auth="true" Jul 08 18:45:37.539576 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.533962 2570 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 08 18:45:37.539576 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.533969 2570 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 08 18:45:37.539576 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.533974 2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 08 18:45:37.539576 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.533980 2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 08 18:45:37.539576 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.533987 2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 08 18:45:37.539576 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.533995 2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 08 18:45:37.539576 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534000 2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 08 18:45:37.539576 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534005 2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 08 18:45:37.539576 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534011 2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 08 18:45:37.539576 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534016 2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 08 18:45:37.539576 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534021 2570 flags.go:64] FLAG: --cgroup-root="" Jul 08 18:45:37.539576 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534026 2570 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 08 18:45:37.539576 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534031 2570 flags.go:64] FLAG: --client-ca-file="" Jul 08 18:45:37.539576 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534036 2570 flags.go:64] FLAG: --cloud-provider="external" Jul 08 18:45:37.539576 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534041 2570 flags.go:64] FLAG: --cluster-dns="[]" Jul 08 18:45:37.539576 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534047 2570 flags.go:64] FLAG: --cluster-domain="" Jul 08 18:45:37.539576 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534052 2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 08 18:45:37.539576 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534057 2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 08 18:45:37.539576 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534062 2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 08 18:45:37.539576 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534068 2570 flags.go:64] FLAG: --container-log-max-files="5" Jul 08 18:45:37.539576 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534074 2570 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 08 18:45:37.539576 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534079 2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 08 18:45:37.540129 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534084 2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 08 18:45:37.540129 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534089 2570 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 08 18:45:37.540129 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534094 2570 flags.go:64] FLAG: --contention-profiling="false" Jul 08 18:45:37.540129 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534099 2570 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 08 18:45:37.540129 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534103 2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 08 18:45:37.540129 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534111 2570 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 08 18:45:37.540129 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534116 2570 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 08 18:45:37.540129 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534123 2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 08 18:45:37.540129 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534128 2570 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 08 18:45:37.540129 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534133 2570 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 08 18:45:37.540129 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534138 2570 flags.go:64] FLAG: --enable-load-reader="false" Jul 08 18:45:37.540129 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534143 2570 flags.go:64] FLAG: --enable-server="true" Jul 08 18:45:37.540129 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534148 2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 08 18:45:37.540129 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534156 2570 flags.go:64] FLAG: --event-burst="100" Jul 08 18:45:37.540129 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534160 2570 flags.go:64] FLAG: --event-qps="50" Jul 08 18:45:37.540129 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534165 2570 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 08 18:45:37.540129 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534171 2570 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 08 18:45:37.540129 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534175 2570 flags.go:64] FLAG: --eviction-hard="" Jul 08 18:45:37.540129 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534181 2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 08 18:45:37.540129 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534186 2570 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 08 18:45:37.540129 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534191 2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 08 18:45:37.540129 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534195 2570 flags.go:64] FLAG: --eviction-soft="" Jul 08 18:45:37.540129 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534201 2570 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 08 18:45:37.540129 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534206 2570 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 08 18:45:37.540129 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534210 2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 08 18:45:37.540749 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534215 2570 flags.go:64] FLAG: --experimental-mounter-path="" Jul 08 18:45:37.540749 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534220 2570 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 08 18:45:37.540749 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534225 2570 flags.go:64] FLAG: --fail-swap-on="true" Jul 08 18:45:37.540749 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534229 2570 flags.go:64] FLAG: --feature-gates="" Jul 08 18:45:37.540749 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534236 2570 flags.go:64] FLAG: --file-check-frequency="20s" Jul 08 18:45:37.540749 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534241 2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 08 18:45:37.540749 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534246 2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 08 18:45:37.540749 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534251 2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 08 18:45:37.540749 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534257 2570 flags.go:64] FLAG: --healthz-port="10248" Jul 08 18:45:37.540749 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534262 2570 flags.go:64] FLAG: --help="false" Jul 08 18:45:37.540749 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534267 2570 flags.go:64] FLAG: --hostname-override="ip-10-0-130-48.ec2.internal" Jul 08 18:45:37.540749 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534272 2570 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 08 18:45:37.540749 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534279 2570 flags.go:64] FLAG: --http-check-frequency="20s" Jul 08 18:45:37.540749 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534284 2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 08 18:45:37.540749 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534290 2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 08 18:45:37.540749 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534296 2570 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 08 18:45:37.540749 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534300 2570 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 08 18:45:37.540749 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534320 2570 flags.go:64] FLAG: --image-service-endpoint="" Jul 08 18:45:37.540749 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534325 2570 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 08 18:45:37.540749 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534330 2570 flags.go:64] FLAG: --kube-api-burst="100" Jul 08 18:45:37.540749 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534335 2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 08 18:45:37.540749 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534341 2570 flags.go:64] FLAG: --kube-api-qps="50" Jul 08 18:45:37.540749 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534345 2570 flags.go:64] FLAG: --kube-reserved="" Jul 08 18:45:37.540749 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534350 2570 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 08 18:45:37.541376 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534356 2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 08 18:45:37.541376 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534362 2570 flags.go:64] FLAG: --kubelet-cgroups="" Jul 08 18:45:37.541376 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534367 2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 08 18:45:37.541376 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534372 2570 flags.go:64] FLAG: --lock-file="" Jul 08 18:45:37.541376 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534377 2570 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 08 18:45:37.541376 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534382 2570 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 08 18:45:37.541376 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534387 2570 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 08 18:45:37.541376 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534396 2570 flags.go:64] FLAG: --log-json-split-stream="false" Jul 08 18:45:37.541376 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534401 2570 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 08 18:45:37.541376 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534405 2570 flags.go:64] FLAG: --log-text-split-stream="false" Jul 08 18:45:37.541376 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534410 2570 flags.go:64] FLAG: --logging-format="text" Jul 08 18:45:37.541376 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534415 2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 08 18:45:37.541376 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534420 2570 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 08 18:45:37.541376 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534425 2570 flags.go:64] FLAG: --manifest-url="" Jul 08 18:45:37.541376 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534430 2570 flags.go:64] FLAG: --manifest-url-header="" Jul 08 18:45:37.541376 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534437 2570 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 08 18:45:37.541376 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534443 2570 flags.go:64] FLAG: --max-open-files="1000000" Jul 08 18:45:37.541376 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534449 2570 flags.go:64] FLAG: --max-pods="110" Jul 08 18:45:37.541376 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534454 2570 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 08 18:45:37.541376 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534459 2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 08 18:45:37.541376 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534466 2570 flags.go:64] FLAG: --memory-manager-policy="None" Jul 08 18:45:37.541376 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534471 2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 08 18:45:37.541376 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534476 2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 08 18:45:37.541376 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534480 2570 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 08 18:45:37.541376 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534485 2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 08 18:45:37.542020 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534497 2570 flags.go:64] FLAG: --node-status-max-images="50" Jul 08 18:45:37.542020 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534501 2570 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 08 18:45:37.542020 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534506 2570 flags.go:64] FLAG: --oom-score-adj="-999" Jul 08 18:45:37.542020 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534511 2570 flags.go:64] FLAG: --pod-cidr="" Jul 08 18:45:37.542020 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534516 2570 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 08 18:45:37.542020 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534525 2570 flags.go:64] FLAG: --pod-manifest-path="" Jul 08 18:45:37.542020 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534530 2570 flags.go:64] FLAG: --pod-max-pids="-1" Jul 08 18:45:37.542020 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534535 2570 flags.go:64] FLAG: --pods-per-core="0" Jul 08 18:45:37.542020 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534540 2570 flags.go:64] FLAG: --port="10250" Jul 08 18:45:37.542020 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534545 2570 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 08 18:45:37.542020 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534550 2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06355f818c05afbe0" Jul 08 18:45:37.542020 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534556 2570 flags.go:64] FLAG: --qos-reserved="" Jul 08 18:45:37.542020 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534560 2570 flags.go:64] FLAG: --read-only-port="10255" Jul 08 18:45:37.542020 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534565 2570 flags.go:64] FLAG: --register-node="true" Jul 08 18:45:37.542020 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534570 2570 flags.go:64] FLAG: --register-with-taints="" Jul 08 18:45:37.542020 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534576 2570 flags.go:64] FLAG: --registry-burst="10" Jul 08 18:45:37.542020 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534580 2570 flags.go:64] FLAG: --registry-qps="5" Jul 08 18:45:37.542020 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534585 2570 flags.go:64] FLAG: --reserved-cpus="" Jul 08 18:45:37.542020 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534590 2570 flags.go:64] FLAG: --reserved-memory="" Jul 08 18:45:37.542020 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534597 2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 08 18:45:37.542020 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534602 2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 08 18:45:37.542020 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534607 2570 flags.go:64] FLAG: --rotate-certificates="false" Jul 08 18:45:37.542020 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534612 2570 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 08 18:45:37.542020 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534617 2570 flags.go:64] FLAG: --runonce="false" Jul 08 18:45:37.542020 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534621 2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 08 18:45:37.542650 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534626 2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 08 18:45:37.542650 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534631 2570 flags.go:64] FLAG: --seccomp-default="false" Jul 08 18:45:37.542650 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534638 2570 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 08 18:45:37.542650 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534643 2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 08 18:45:37.542650 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534648 2570 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 08 18:45:37.542650 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534653 2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 08 18:45:37.542650 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534658 2570 flags.go:64] FLAG: --storage-driver-password="root" Jul 08 18:45:37.542650 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534664 2570 flags.go:64] FLAG: --storage-driver-secure="false" Jul 08 18:45:37.542650 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534668 2570 flags.go:64] FLAG: --storage-driver-table="stats" Jul 08 18:45:37.542650 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534673 2570 flags.go:64] FLAG: --storage-driver-user="root" Jul 08 18:45:37.542650 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534677 2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 08 18:45:37.542650 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534683 2570 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 08 18:45:37.542650 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534688 2570 flags.go:64] FLAG: --system-cgroups="" Jul 08 18:45:37.542650 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534692 2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 08 18:45:37.542650 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534701 2570 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 08 18:45:37.542650 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534705 2570 flags.go:64] FLAG: --tls-cert-file="" Jul 08 18:45:37.542650 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534711 2570 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 08 18:45:37.542650 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534717 2570 flags.go:64] FLAG: --tls-min-version="" Jul 08 18:45:37.542650 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534722 2570 flags.go:64] FLAG: --tls-private-key-file="" Jul 08 18:45:37.542650 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534727 2570 flags.go:64] FLAG: --topology-manager-policy="none" Jul 08 18:45:37.542650 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534732 2570 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 08 18:45:37.542650 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534736 2570 flags.go:64] FLAG: --topology-manager-scope="container" Jul 08 18:45:37.542650 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534741 2570 flags.go:64] FLAG: --v="2" Jul 08 18:45:37.542650 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534748 2570 flags.go:64] FLAG: --version="false" Jul 08 18:45:37.542650 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534758 2570 flags.go:64] FLAG: --vmodule="" Jul 08 18:45:37.543272 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534764 2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 08 18:45:37.543272 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.534769 2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 08 18:45:37.543272 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.534956 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 18:45:37.543272 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.534964 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 18:45:37.543272 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.534969 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 18:45:37.543272 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.534974 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 18:45:37.543272 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.534978 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 18:45:37.543272 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.534983 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 18:45:37.543272 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.534988 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 18:45:37.543272 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.534993 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 18:45:37.543272 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.534998 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 18:45:37.543272 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535002 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 18:45:37.543272 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535006 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 18:45:37.543272 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535010 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 18:45:37.543272 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535017 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 18:45:37.543272 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535023 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 18:45:37.543272 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535028 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 18:45:37.543272 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535033 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 18:45:37.543272 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535037 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 18:45:37.543272 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535042 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 18:45:37.543771 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535047 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 18:45:37.543771 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535051 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 18:45:37.543771 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535055 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 18:45:37.543771 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535059 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 18:45:37.543771 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535065 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 18:45:37.543771 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535069 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 18:45:37.543771 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535074 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 18:45:37.543771 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535078 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 18:45:37.543771 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535082 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 18:45:37.543771 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535087 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 18:45:37.543771 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535091 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 18:45:37.543771 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535097 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 18:45:37.543771 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535102 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 18:45:37.543771 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535106 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 18:45:37.543771 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535111 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 18:45:37.543771 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535115 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 18:45:37.543771 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535120 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 18:45:37.543771 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535124 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 18:45:37.543771 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535128 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 18:45:37.543771 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535133 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 18:45:37.544248 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535139 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 18:45:37.544248 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535144 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 18:45:37.544248 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535149 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 18:45:37.544248 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535154 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 18:45:37.544248 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535159 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 18:45:37.544248 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535164 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 18:45:37.544248 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535169 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 18:45:37.544248 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535174 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 18:45:37.544248 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535178 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 18:45:37.544248 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535182 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 18:45:37.544248 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535187 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 18:45:37.544248 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535191 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 18:45:37.544248 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535195 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 18:45:37.544248 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535200 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 18:45:37.544248 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535204 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 18:45:37.544248 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535209 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 18:45:37.544248 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535213 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 18:45:37.544248 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535222 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 18:45:37.544248 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535227 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 18:45:37.544763 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535231 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 18:45:37.544763 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535235 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 18:45:37.544763 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535239 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 18:45:37.544763 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535244 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 18:45:37.544763 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535250 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 18:45:37.544763 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535254 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 18:45:37.544763 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535259 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 18:45:37.544763 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535263 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 18:45:37.544763 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535267 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 18:45:37.544763 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535272 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 18:45:37.544763 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535276 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 18:45:37.544763 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535280 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 18:45:37.544763 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535285 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 18:45:37.544763 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535289 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 18:45:37.544763 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535293 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 18:45:37.544763 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535297 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 18:45:37.544763 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535301 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 08 18:45:37.544763 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535321 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 18:45:37.544763 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535325 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 18:45:37.544763 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535329 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 18:45:37.545237 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535333 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 18:45:37.545237 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535337 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 18:45:37.545237 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535340 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 18:45:37.545237 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535344 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 18:45:37.545237 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535348 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 18:45:37.545237 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535352 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 18:45:37.545237 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535356 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 18:45:37.545237 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535360 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 18:45:37.545237 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535364 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 18:45:37.545237 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535368 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 18:45:37.545237 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535374 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 18:45:37.545237 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535400 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 18:45:37.545237 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535423 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 18:45:37.545237 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535427 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 18:45:37.545237 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535430 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 18:45:37.545237 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535434 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 18:45:37.545237 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535438 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 18:45:37.545237 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535441 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 18:45:37.545237 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535444 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 18:45:37.545703 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535447 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 18:45:37.545703 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535450 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 18:45:37.545703 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.535453 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 18:45:37.545703 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.535955 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 18:45:37.545703 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.542884 2570 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 08 18:45:37.545703 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.542905 2570 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 08 18:45:37.545703 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.542925 2570 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 08 18:45:37.545703 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.542931 2570 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 08 18:45:37.545703 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.542985 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 18:45:37.545703 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.542991 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 18:45:37.545703 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.542995 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 18:45:37.545703 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.542998 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 18:45:37.545703 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543001 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 18:45:37.545703 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543004 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 18:45:37.545703 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543007 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 18:45:37.546075 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543010 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 18:45:37.546075 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543012 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 18:45:37.546075 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543015 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 18:45:37.546075 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543020 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 18:45:37.546075 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543024 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 18:45:37.546075 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543028 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 18:45:37.546075 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543032 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 18:45:37.546075 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543036 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 18:45:37.546075 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543039 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 18:45:37.546075 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543042 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 18:45:37.546075 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543045 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 18:45:37.546075 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543047 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 18:45:37.546075 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543050 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 18:45:37.546075 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543053 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 18:45:37.546075 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543056 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 18:45:37.546075 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543058 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 18:45:37.546075 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543061 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 18:45:37.546075 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543064 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 18:45:37.546075 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543066 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 08 18:45:37.546545 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543069 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 18:45:37.546545 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543072 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 18:45:37.546545 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543075 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 18:45:37.546545 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543079 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 18:45:37.546545 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543082 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 18:45:37.546545 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543085 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 18:45:37.546545 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543088 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 18:45:37.546545 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543090 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 18:45:37.546545 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543093 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 18:45:37.546545 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543096 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 18:45:37.546545 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543099 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 18:45:37.546545 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543102 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 18:45:37.546545 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543105 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 18:45:37.546545 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543107 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 18:45:37.546545 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543110 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 18:45:37.546545 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543114 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 18:45:37.546545 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543118 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 18:45:37.546545 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543121 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 18:45:37.546545 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543124 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 18:45:37.546987 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543126 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 18:45:37.546987 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543129 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 18:45:37.546987 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543132 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 18:45:37.546987 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543135 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 18:45:37.546987 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543137 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 18:45:37.546987 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543140 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 18:45:37.546987 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543143 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 18:45:37.546987 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543145 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 18:45:37.546987 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543148 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 18:45:37.546987 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543151 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 18:45:37.546987 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543154 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 18:45:37.546987 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543157 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 18:45:37.546987 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543159 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 18:45:37.546987 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543162 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 18:45:37.546987 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543165 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 18:45:37.546987 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543168 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 18:45:37.546987 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543172 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 18:45:37.546987 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543175 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 18:45:37.546987 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543177 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 18:45:37.546987 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543180 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 18:45:37.547484 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543183 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 18:45:37.547484 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543186 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 18:45:37.547484 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543188 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 18:45:37.547484 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543191 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 18:45:37.547484 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543194 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 18:45:37.547484 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543196 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 18:45:37.547484 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543199 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 18:45:37.547484 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543201 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 18:45:37.547484 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543204 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 18:45:37.547484 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543206 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 18:45:37.547484 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543209 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 18:45:37.547484 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543211 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 18:45:37.547484 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543214 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 18:45:37.547484 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543217 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 18:45:37.547484 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543219 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 18:45:37.547484 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543222 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 18:45:37.547484 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543224 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 18:45:37.547484 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543227 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 18:45:37.547484 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543230 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 18:45:37.547929 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543232 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 18:45:37.547929 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543235 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 18:45:37.547929 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543238 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 18:45:37.547929 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543240 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 18:45:37.547929 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543243 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 18:45:37.547929 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543246 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 18:45:37.547929 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543248 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 18:45:37.547929 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543251 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 18:45:37.547929 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543253 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 18:45:37.547929 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543257 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 18:45:37.547929 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543260 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 18:45:37.547929 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543263 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 18:45:37.547929 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543266 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 18:45:37.547929 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543268 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 18:45:37.547929 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543271 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 18:45:37.548294 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.543276 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 18:45:37.548294 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543410 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 18:45:37.548294 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543415 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 18:45:37.548294 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543418 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 18:45:37.548294 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543421 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 18:45:37.548294 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543424 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 18:45:37.548294 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543427 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 18:45:37.548294 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543429 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 18:45:37.548294 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543433 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 18:45:37.548294 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543437 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 18:45:37.548294 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543440 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 18:45:37.548294 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543442 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 18:45:37.548294 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543445 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 18:45:37.548294 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543447 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 18:45:37.548294 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543450 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 18:45:37.548699 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543453 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 18:45:37.548699 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543455 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 18:45:37.548699 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543458 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 18:45:37.548699 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543460 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 18:45:37.548699 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543463 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 18:45:37.548699 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543467 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 18:45:37.548699 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543470 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 18:45:37.548699 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543473 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 18:45:37.548699 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543476 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 18:45:37.548699 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543479 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 18:45:37.548699 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543481 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 18:45:37.548699 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543485 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 18:45:37.548699 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543488 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 18:45:37.548699 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543490 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 18:45:37.548699 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543493 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 18:45:37.548699 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543496 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 18:45:37.548699 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543499 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 08 18:45:37.548699 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543501 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 18:45:37.548699 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543504 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 18:45:37.549154 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543506 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 18:45:37.549154 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543509 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 18:45:37.549154 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543511 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 18:45:37.549154 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543514 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 18:45:37.549154 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543516 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 18:45:37.549154 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543519 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 18:45:37.549154 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543521 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 18:45:37.549154 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543524 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 18:45:37.549154 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543527 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 18:45:37.549154 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543529 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 18:45:37.549154 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543532 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 18:45:37.549154 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543536 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 18:45:37.549154 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543539 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 18:45:37.549154 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543542 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 18:45:37.549154 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543545 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 18:45:37.549154 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543548 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 18:45:37.549154 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543550 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 18:45:37.549154 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543553 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 18:45:37.549154 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543555 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 18:45:37.549617 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543558 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 18:45:37.549617 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543560 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 18:45:37.549617 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543563 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 18:45:37.549617 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543565 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 18:45:37.549617 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543568 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 18:45:37.549617 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543570 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 18:45:37.549617 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543573 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 18:45:37.549617 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543576 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 18:45:37.549617 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543579 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 18:45:37.549617 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543582 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 18:45:37.549617 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543585 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 18:45:37.549617 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543588 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 18:45:37.549617 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543591 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 18:45:37.549617 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543593 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 18:45:37.549617 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543596 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 18:45:37.549617 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543599 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 18:45:37.549617 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543601 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 18:45:37.549617 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543603 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 18:45:37.549617 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543606 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 18:45:37.550072 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543608 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 18:45:37.550072 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543611 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 18:45:37.550072 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543613 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 18:45:37.550072 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543616 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 18:45:37.550072 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543619 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 18:45:37.550072 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543621 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 18:45:37.550072 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543624 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 18:45:37.550072 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543626 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 18:45:37.550072 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543629 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 18:45:37.550072 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543631 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 18:45:37.550072 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543634 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 18:45:37.550072 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543636 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 18:45:37.550072 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543639 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 18:45:37.550072 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543641 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 18:45:37.550072 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543644 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 18:45:37.550072 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543646 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 18:45:37.550072 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543649 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 18:45:37.550072 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543651 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 18:45:37.550072 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543654 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 18:45:37.550072 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543657 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 18:45:37.550569 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543659 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 18:45:37.550569 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543662 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 18:45:37.550569 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543664 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 18:45:37.550569 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543667 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 18:45:37.550569 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543670 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 18:45:37.550569 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543672 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 18:45:37.550569 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543675 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 18:45:37.550569 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:37.543677 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 18:45:37.550569 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.543682 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 18:45:37.550569 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.543813 2570 server.go:962] "Client rotation is on, will bootstrap in background" Jul 08 18:45:37.550569 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.545935 2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 08 18:45:37.550569 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.547464 2570 server.go:1019] "Starting client certificate rotation" Jul 08 18:45:37.550569 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.547570 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 18:45:37.550569 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.547603 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 18:45:37.551973 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.551802 2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 18:45:37.553677 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.553659 2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 18:45:37.566000 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.565983 2570 log.go:25] "Validated CRI v1 runtime API" Jul 08 18:45:37.568171 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.568155 2570 log.go:25] "Validated CRI v1 image API" Jul 08 18:45:37.569036 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.569012 2570 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 08 18:45:37.571342 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.571322 2570 fs.go:135] Filesystem UUIDs: map[68ae6eac-88bc-4426-816a-b6f224901554:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 83eb4a3d-8ac0-430b-a9da-faf903036642:/dev/nvme0n1p3] Jul 08 18:45:37.571414 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.571341 2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 08 18:45:37.577244 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.577119 2570 manager.go:216] Machine: {Timestamp:2026-07-08 18:45:37.575901741 +0000 UTC m=+0.319121943 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100392 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec292c1bf5cdf9d6773c399005c573e8 SystemUUID:ec292c1b-f5cd-f9d6-773c-399005c573e8 BootID:db89f1b2-b292-4898-9bae-651af39c02ee Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e7:2c:90:31:03 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e7:2c:90:31:03 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:d0:3a:4c:24:ee Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 08 18:45:37.577244 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.577234 2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 08 18:45:37.577426 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.577366 2570 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 08 18:45:37.579140 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.579116 2570 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 08 18:45:37.579287 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.579142 2570 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-48.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 08 18:45:37.579777 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.579767 2570 topology_manager.go:138] "Creating topology manager with none policy" Jul 08 18:45:37.579813 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.579780 2570 container_manager_linux.go:309] "Creating device plugin manager" Jul 08 18:45:37.579813 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.579787 2570 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 18:45:37.580410 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.580400 2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 18:45:37.582044 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.582030 2570 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 08 18:45:37.585281 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.585269 2570 state_mem.go:36] "Initialized new in-memory state store" Jul 08 18:45:37.585405 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.585395 2570 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 08 18:45:37.586267 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.586258 2570 kubelet.go:486] "Attempting to sync node with API server" Jul 08 18:45:37.586317 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.586273 2570 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 08 18:45:37.586317 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.586288 2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 08 18:45:37.586317 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.586296 2570 kubelet.go:398] "Adding apiserver pod source" Jul 08 18:45:37.586445 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.586325 2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 08 18:45:37.586694 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.586682 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 18:45:37.586731 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.586705 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 18:45:37.589024 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.588986 2570 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 08 18:45:37.591952 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.591929 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 18:45:37.592202 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.592187 2570 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 08 18:45:37.592241 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.592219 2570 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 08 18:45:37.593235 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.593223 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 08 18:45:37.593289 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.593240 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 08 18:45:37.593289 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.593247 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 08 18:45:37.593289 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.593252 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 08 18:45:37.593289 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.593262 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 08 18:45:37.593289 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.593268 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 08 18:45:37.593289 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.593277 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 08 18:45:37.593289 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.593283 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 08 18:45:37.593289 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.593290 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 08 18:45:37.593522 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.593295 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 08 18:45:37.593522 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.593319 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 08 18:45:37.593522 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.593329 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 08 18:45:37.594201 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.594191 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 08 18:45:37.594201 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.594200 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 08 18:45:37.597739 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.597724 2570 server.go:1268] "Started kubelet" Jul 08 18:45:37.597860 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.597805 2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 08 18:45:37.597955 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.597861 2570 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 08 18:45:37.597955 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.597883 2570 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 08 18:45:37.598646 ip-10-0-130-48 systemd[1]: Started Kubernetes Kubelet. Jul 08 18:45:37.599012 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.598979 2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-48.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 08 18:45:37.599187 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:37.599084 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 08 18:45:37.599247 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:37.599190 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-48.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 08 18:45:37.600128 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.600108 2570 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 08 18:45:37.600450 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.600433 2570 server.go:310] "Adding debug handlers to kubelet server" Jul 08 18:45:37.602418 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:37.602395 2570 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 08 18:45:37.606982 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:37.606083 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-48.ec2.internal.18c064d97682fb03 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-48.ec2.internal,UID:ip-10-0-130-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-48.ec2.internal,},FirstTimestamp:2026-07-08 18:45:37.597692675 +0000 UTC m=+0.340912876,LastTimestamp:2026-07-08 18:45:37.597692675 +0000 UTC m=+0.340912876,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-48.ec2.internal,}" Jul 08 18:45:37.607811 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.607797 2570 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 08 18:45:37.607880 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.607803 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 08 18:45:37.607932 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:37.607893 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-48.ec2.internal\" not found" Jul 08 18:45:37.608575 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.608557 2570 factory.go:55] Registering systemd factory Jul 08 18:45:37.608575 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.608575 2570 factory.go:223] Registration of the systemd container factory successfully Jul 08 18:45:37.608762 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.608739 2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 08 18:45:37.609015 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.608877 2570 factory.go:153] Registering CRI-O factory Jul 08 18:45:37.609015 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.608910 2570 factory.go:223] Registration of the crio container factory successfully Jul 08 18:45:37.609109 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.609051 2570 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 08 18:45:37.609109 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.609065 2570 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 08 18:45:37.609109 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.609074 2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 08 18:45:37.609109 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.609101 2570 factory.go:103] Registering Raw factory Jul 08 18:45:37.609300 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.609119 2570 manager.go:1196] Started watching for new ooms in manager Jul 08 18:45:37.609300 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.609210 2570 reconstruct.go:97] "Volume reconstruction finished" Jul 08 18:45:37.609300 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.609219 2570 reconciler.go:29] "Reconciler: start to sync state" Jul 08 18:45:37.609842 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.609824 2570 manager.go:349] Starting recovery of all containers Jul 08 18:45:37.610569 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:37.610516 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 08 18:45:37.610665 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:37.610643 2570 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-48.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 08 18:45:37.622380 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.622242 2570 manager.go:354] Recovery completed Jul 08 18:45:37.626564 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.626550 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 18:45:37.628529 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.628497 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-48.ec2.internal" event="NodeHasSufficientMemory" Jul 08 18:45:37.628597 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.628549 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-48.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 18:45:37.628597 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.628565 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-48.ec2.internal" event="NodeHasSufficientPID" Jul 08 18:45:37.629240 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.629224 2570 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 08 18:45:37.629240 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.629240 2570 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 08 18:45:37.629342 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.629257 2570 state_mem.go:36] "Initialized new in-memory state store" Jul 08 18:45:37.629342 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.629226 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 18:45:37.630023 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.630006 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-48.ec2.internal" event="NodeHasSufficientMemory" Jul 08 18:45:37.630098 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.630038 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-48.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 18:45:37.630098 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.630051 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-48.ec2.internal" event="NodeHasSufficientPID" Jul 08 18:45:37.630442 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.630427 2570 policy_none.go:49] "None policy: Start" Jul 08 18:45:37.630510 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.630453 2570 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 08 18:45:37.630510 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.630467 2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 08 18:45:37.631600 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.631587 2570 policy_none.go:47] "Start" Jul 08 18:45:37.641066 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:37.640992 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-48.ec2.internal.18c064d978598e95 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-48.ec2.internal,UID:ip-10-0-130-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-48.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-48.ec2.internal,},FirstTimestamp:2026-07-08 18:45:37.628532373 +0000 UTC m=+0.371752581,LastTimestamp:2026-07-08 18:45:37.628532373 +0000 UTC m=+0.371752581,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-48.ec2.internal,}" Jul 08 18:45:37.641530 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.641504 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d2nvv" Jul 08 18:45:37.645970 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:37.645883 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-48.ec2.internal.18c064d97859ee23 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-48.ec2.internal,UID:ip-10-0-130-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-48.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-48.ec2.internal,},FirstTimestamp:2026-07-08 18:45:37.628556835 +0000 UTC m=+0.371777246,LastTimestamp:2026-07-08 18:45:37.628556835 +0000 UTC m=+0.371777246,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-48.ec2.internal,}" Jul 08 18:45:37.650430 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:37.650366 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-48.ec2.internal.18c064d9785a270b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-48.ec2.internal,UID:ip-10-0-130-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-48.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-48.ec2.internal,},FirstTimestamp:2026-07-08 18:45:37.628571403 +0000 UTC m=+0.371791605,LastTimestamp:2026-07-08 18:45:37.628571403 +0000 UTC m=+0.371791605,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-48.ec2.internal,}" Jul 08 18:45:37.652802 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.652783 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d2nvv" Jul 08 18:45:37.674898 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.674883 2570 manager.go:337] "Starting Device Plugin manager" Jul 08 18:45:37.674988 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:37.674920 2570 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 08 18:45:37.674988 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.674930 2570 server.go:87] "Starting device plugin registration server" Jul 08 18:45:37.675196 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.675185 2570 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 08 18:45:37.675239 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.675198 2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 08 18:45:37.675295 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.675281 2570 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 08 18:45:37.675393 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.675383 2570 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 08 18:45:37.675432 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.675394 2570 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 08 18:45:37.676033 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:37.676008 2570 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 08 18:45:37.676108 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:37.676072 2570 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-48.ec2.internal\" not found" Jul 08 18:45:37.775875 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.775801 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 18:45:37.777219 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.777203 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-48.ec2.internal" event="NodeHasSufficientMemory" Jul 08 18:45:37.777282 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.777234 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-48.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 18:45:37.777282 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.777246 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-48.ec2.internal" event="NodeHasSufficientPID" Jul 08 18:45:37.777282 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.777276 2570 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-48.ec2.internal" Jul 08 18:45:37.785747 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.785717 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 08 18:45:37.786865 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.786840 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 08 18:45:37.786936 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.786890 2570 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 08 18:45:37.786936 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.786924 2570 kubelet.go:2448] "Starting kubelet main sync loop" Jul 08 18:45:37.787013 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:37.786965 2570 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 08 18:45:37.787856 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.787840 2570 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-48.ec2.internal" Jul 08 18:45:37.787946 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:37.787865 2570 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-48.ec2.internal\": node \"ip-10-0-130-48.ec2.internal\" not found" Jul 08 18:45:37.789079 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.789058 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 18:45:37.842008 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:37.841976 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-48.ec2.internal\" not found" Jul 08 18:45:37.888018 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.887987 2570 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-48.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-48.ec2.internal"] Jul 08 18:45:37.888092 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.888084 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 18:45:37.889057 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.889041 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-48.ec2.internal" event="NodeHasSufficientMemory" Jul 08 18:45:37.889111 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.889077 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-48.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 18:45:37.889111 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.889092 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-48.ec2.internal" event="NodeHasSufficientPID" Jul 08 18:45:37.890485 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.890473 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 18:45:37.890613 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.890599 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-48.ec2.internal" Jul 08 18:45:37.890650 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.890637 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 18:45:37.892939 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.892919 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-48.ec2.internal" event="NodeHasSufficientMemory" Jul 08 18:45:37.893044 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.892952 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-48.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 18:45:37.893044 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.892966 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-48.ec2.internal" event="NodeHasSufficientPID" Jul 08 18:45:37.893044 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.892975 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-48.ec2.internal" event="NodeHasSufficientMemory" Jul 08 18:45:37.893044 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.892997 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-48.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 18:45:37.893044 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.893007 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-48.ec2.internal" event="NodeHasSufficientPID" Jul 08 18:45:37.894239 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.894227 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-48.ec2.internal" Jul 08 18:45:37.894279 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.894250 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 18:45:37.894992 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.894975 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-48.ec2.internal" event="NodeHasSufficientMemory" Jul 08 18:45:37.895076 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.895004 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-48.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 18:45:37.895076 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.895019 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-48.ec2.internal" event="NodeHasSufficientPID" Jul 08 18:45:37.909909 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:37.909886 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-48.ec2.internal\" not found" node="ip-10-0-130-48.ec2.internal" Jul 08 18:45:37.910586 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.910570 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d7ba020ad0d6e8fc7260c108455cdc7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-48.ec2.internal\" (UID: \"3d7ba020ad0d6e8fc7260c108455cdc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-48.ec2.internal" Jul 08 18:45:37.910640 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.910594 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/258c89e0ffadf0a34366b9c773cb3f43-config\") pod \"kube-apiserver-proxy-ip-10-0-130-48.ec2.internal\" (UID: \"258c89e0ffadf0a34366b9c773cb3f43\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-48.ec2.internal" Jul 08 18:45:37.910640 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:37.910613 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3d7ba020ad0d6e8fc7260c108455cdc7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-48.ec2.internal\" (UID: \"3d7ba020ad0d6e8fc7260c108455cdc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-48.ec2.internal" Jul 08 18:45:37.914327 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:37.914299 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-48.ec2.internal\" not found" node="ip-10-0-130-48.ec2.internal" Jul 08 18:45:37.942814 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:37.942784 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-48.ec2.internal\" not found" Jul 08 18:45:38.011551 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:38.011523 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/258c89e0ffadf0a34366b9c773cb3f43-config\") pod \"kube-apiserver-proxy-ip-10-0-130-48.ec2.internal\" (UID: \"258c89e0ffadf0a34366b9c773cb3f43\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-48.ec2.internal" Jul 08 18:45:38.011655 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:38.011554 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3d7ba020ad0d6e8fc7260c108455cdc7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-48.ec2.internal\" (UID: \"3d7ba020ad0d6e8fc7260c108455cdc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-48.ec2.internal" Jul 08 18:45:38.011655 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:38.011572 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d7ba020ad0d6e8fc7260c108455cdc7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-48.ec2.internal\" (UID: \"3d7ba020ad0d6e8fc7260c108455cdc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-48.ec2.internal" Jul 08 18:45:38.011655 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:38.011616 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d7ba020ad0d6e8fc7260c108455cdc7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-48.ec2.internal\" (UID: \"3d7ba020ad0d6e8fc7260c108455cdc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-48.ec2.internal" Jul 08 18:45:38.011655 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:38.011645 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/258c89e0ffadf0a34366b9c773cb3f43-config\") pod \"kube-apiserver-proxy-ip-10-0-130-48.ec2.internal\" (UID: \"258c89e0ffadf0a34366b9c773cb3f43\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-48.ec2.internal" Jul 08 18:45:38.011795 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:38.011645 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3d7ba020ad0d6e8fc7260c108455cdc7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-48.ec2.internal\" (UID: \"3d7ba020ad0d6e8fc7260c108455cdc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-48.ec2.internal" Jul 08 18:45:38.043419 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:38.043361 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-48.ec2.internal\" not found" Jul 08 18:45:38.144231 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:38.144187 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-48.ec2.internal\" not found" Jul 08 18:45:38.213704 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:38.213677 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-48.ec2.internal" Jul 08 18:45:38.219036 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:38.219009 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-48.ec2.internal" Jul 08 18:45:38.245210 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:38.245182 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-48.ec2.internal\" not found" Jul 08 18:45:38.345850 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:38.345746 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-48.ec2.internal\" not found" Jul 08 18:45:38.446231 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:38.446194 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-48.ec2.internal\" not found" Jul 08 18:45:38.546806 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:38.546774 2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 08 18:45:38.546806 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:38.546777 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-48.ec2.internal\" not found" Jul 08 18:45:38.547367 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:38.546902 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 08 18:45:38.647099 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:38.647076 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-48.ec2.internal\" not found" Jul 08 18:45:38.654926 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:38.654873 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-07 18:40:39 +0000 UTC" deadline="2028-04-03 00:31:30.88380038 +0000 UTC" Jul 08 18:45:38.654926 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:38.654924 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15221h45m52.228881923s" Jul 08 18:45:38.656355 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:38.656338 2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 18:45:38.747330 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:38.747295 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-48.ec2.internal\" not found" Jul 08 18:45:38.747442 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:38.747300 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d7ba020ad0d6e8fc7260c108455cdc7.slice/crio-20f2f1d91192442e65ef7891abc1b37aaf8ac5f29944dcaadfeb441755b6155f WatchSource:0}: Error finding container 20f2f1d91192442e65ef7891abc1b37aaf8ac5f29944dcaadfeb441755b6155f: Status 404 returned error can't find the container with id 20f2f1d91192442e65ef7891abc1b37aaf8ac5f29944dcaadfeb441755b6155f Jul 08 18:45:38.747890 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:38.747872 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod258c89e0ffadf0a34366b9c773cb3f43.slice/crio-27cc9461062ef3806043c9158e59d713dba415b19fa0e79433b3d9146eadb846 WatchSource:0}: Error finding container 27cc9461062ef3806043c9158e59d713dba415b19fa0e79433b3d9146eadb846: Status 404 returned error can't find the container with id 27cc9461062ef3806043c9158e59d713dba415b19fa0e79433b3d9146eadb846 Jul 08 18:45:38.751933 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:38.751922 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 18:45:38.766052 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:38.766027 2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 18:45:38.790510 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:38.790469 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-48.ec2.internal" event={"ID":"258c89e0ffadf0a34366b9c773cb3f43","Type":"ContainerStarted","Data":"27cc9461062ef3806043c9158e59d713dba415b19fa0e79433b3d9146eadb846"} Jul 08 18:45:38.791512 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:38.791492 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-48.ec2.internal" event={"ID":"3d7ba020ad0d6e8fc7260c108455cdc7","Type":"ContainerStarted","Data":"20f2f1d91192442e65ef7891abc1b37aaf8ac5f29944dcaadfeb441755b6155f"} Jul 08 18:45:38.797661 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:38.797647 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 18:45:38.798731 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:38.798714 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-48.ec2.internal" event="NodeHasSufficientMemory" Jul 08 18:45:38.798801 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:38.798748 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-48.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 18:45:38.798801 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:38.798759 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-48.ec2.internal" event="NodeHasSufficientPID" Jul 08 18:45:38.808468 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:38.808446 2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 18:45:40.804991 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:40.804953 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-48.ec2.internal" Jul 08 18:45:40.830787 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:40.830756 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 18:45:40.831885 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:40.831868 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-48.ec2.internal" Jul 08 18:45:40.855874 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:40.855846 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 18:45:41.483935 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.483903 2570 apiserver.go:52] "Watching apiserver" Jul 08 18:45:41.490392 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.490365 2570 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 08 18:45:41.492335 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.492296 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-dqz5x","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-48.ec2.internal","openshift-network-diagnostics/network-check-target-mmgzx","openshift-ovn-kubernetes/ovnkube-node-wqmlf","kube-system/konnectivity-agent-tfrw6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfb5g","openshift-dns/node-resolver-8hxcn","openshift-multus/multus-additional-cni-plugins-fdwjq","openshift-multus/multus-c2vbf","openshift-multus/network-metrics-daemon-mwtbr","openshift-network-operator/iptables-alerter-5njd8","kube-system/global-pull-secret-syncer-9gjk9","kube-system/kube-apiserver-proxy-ip-10-0-130-48.ec2.internal","openshift-cluster-node-tuning-operator/tuned-2jcvm"] Jul 08 18:45:41.494058 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.494034 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8hxcn" Jul 08 18:45:41.496645 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.495392 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5njd8" Jul 08 18:45:41.497523 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.497499 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmgzx" Jul 08 18:45:41.497628 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:41.497592 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mmgzx" podUID="f7e42624-526e-4aaf-904f-3541dc59aeda" Jul 08 18:45:41.497747 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.497728 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4l8nw\"" Jul 08 18:45:41.497824 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.497802 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 08 18:45:41.497922 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.497903 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 08 18:45:41.498106 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.498088 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 08 18:45:41.498499 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.498471 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 08 18:45:41.498645 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.498626 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-k5xft\"" Jul 08 18:45:41.498689 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.498474 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 08 18:45:41.499626 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.499411 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.501353 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.501333 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tfrw6" Jul 08 18:45:41.502575 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.502549 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lph9l\"" Jul 08 18:45:41.503228 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.502870 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 08 18:45:41.503228 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.502918 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 08 18:45:41.503228 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.502870 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 08 18:45:41.503228 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.503142 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 08 18:45:41.503228 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.503159 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 08 18:45:41.503526 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.503316 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfb5g" Jul 08 18:45:41.503526 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.503391 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 08 18:45:41.503947 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.503817 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 08 18:45:41.503947 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.503935 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 08 18:45:41.504406 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.503997 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-nkrbf\"" Jul 08 18:45:41.504406 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.504097 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 08 18:45:41.505585 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.505553 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 08 18:45:41.505768 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.505745 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.505922 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.505903 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tmwpx\"" Jul 08 18:45:41.506095 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.506074 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 08 18:45:41.506095 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.506089 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 08 18:45:41.506549 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.506531 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.507879 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.507860 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 08 18:45:41.508648 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.508135 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fdwjq" Jul 08 18:45:41.508648 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.508147 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 08 18:45:41.508648 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.508231 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 08 18:45:41.508648 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.508518 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 08 18:45:41.508895 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.508885 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8h6vb\"" Jul 08 18:45:41.508956 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.508917 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 08 18:45:41.509056 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.509037 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 08 18:45:41.509150 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.509124 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8rg6t\"" Jul 08 18:45:41.509767 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.509653 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwtbr" Jul 08 18:45:41.509767 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:41.509713 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwtbr" podUID="45fe0fb7-f912-4b72-b298-1dae0ceae220" Jul 08 18:45:41.510570 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.510550 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 08 18:45:41.510669 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.510595 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-62hzz\"" Jul 08 18:45:41.510729 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.510695 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 18:45:41.510926 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.510909 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 08 18:45:41.511248 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.511233 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dqz5x" Jul 08 18:45:41.512818 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.512680 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9gjk9" Jul 08 18:45:41.512818 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:41.512738 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9gjk9" podUID="02e604f5-7b5c-4aec-a685-e380d4b20a3e" Jul 08 18:45:41.513516 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.513475 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 08 18:45:41.513807 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.513746 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lm64s\"" Jul 08 18:45:41.513954 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.513924 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 08 18:45:41.514079 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.514058 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 08 18:45:41.516084 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.516061 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxbtx\" (UniqueName: \"kubernetes.io/projected/f7e42624-526e-4aaf-904f-3541dc59aeda-kube-api-access-zxbtx\") pod \"network-check-target-mmgzx\" (UID: \"f7e42624-526e-4aaf-904f-3541dc59aeda\") " pod="openshift-network-diagnostics/network-check-target-mmgzx" Jul 08 18:45:41.516228 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.516100 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-host-kubelet\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.516228 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.516125 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d8fdda41-866c-40a9-95ff-599de68e4672-agent-certs\") pod \"konnectivity-agent-tfrw6\" (UID: \"d8fdda41-866c-40a9-95ff-599de68e4672\") " pod="kube-system/konnectivity-agent-tfrw6" Jul 08 18:45:41.516228 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.516148 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-cnibin\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.516228 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.516172 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d7c69034-21a4-43da-ab51-09b49ac4a67a-iptables-alerter-script\") pod \"iptables-alerter-5njd8\" (UID: \"d7c69034-21a4-43da-ab51-09b49ac4a67a\") " pod="openshift-network-operator/iptables-alerter-5njd8" Jul 08 18:45:41.516228 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.516196 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd427\" (UniqueName: \"kubernetes.io/projected/d7c69034-21a4-43da-ab51-09b49ac4a67a-kube-api-access-dd427\") pod \"iptables-alerter-5njd8\" (UID: \"d7c69034-21a4-43da-ab51-09b49ac4a67a\") " pod="openshift-network-operator/iptables-alerter-5njd8" Jul 08 18:45:41.516228 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.516219 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5377c392-5ccf-4e87-ba8f-69a4fc166088-device-dir\") pod \"aws-ebs-csi-driver-node-vfb5g\" (UID: \"5377c392-5ccf-4e87-ba8f-69a4fc166088\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfb5g" Jul 08 18:45:41.516542 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.516336 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt22s\" (UniqueName: \"kubernetes.io/projected/f79dc66e-2b96-42c4-9ee9-4d036cc63d3a-kube-api-access-pt22s\") pod \"node-resolver-8hxcn\" (UID: \"f79dc66e-2b96-42c4-9ee9-4d036cc63d3a\") " pod="openshift-dns/node-resolver-8hxcn" Jul 08 18:45:41.516542 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.516363 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-etc-openvswitch\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.516542 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.516379 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-run-ovn\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.516542 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.516393 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-system-cni-dir\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.516542 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.516411 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-host-run-k8s-cni-cncf-io\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.516542 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.516428 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1e1cf566-6a03-4273-a870-5f78662ca68c-multus-daemon-config\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.516542 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.516453 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-host-run-multus-certs\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.516542 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.516481 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5377c392-5ccf-4e87-ba8f-69a4fc166088-sys-fs\") pod \"aws-ebs-csi-driver-node-vfb5g\" (UID: \"5377c392-5ccf-4e87-ba8f-69a4fc166088\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfb5g" Jul 08 18:45:41.516916 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.516544 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-host-run-netns\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.516916 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.516588 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c95d4430-b571-42e1-b016-686b9aa91245-ovnkube-config\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.516916 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.516642 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c95d4430-b571-42e1-b016-686b9aa91245-ovnkube-script-lib\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.516916 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.516668 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-host-var-lib-cni-multus\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.516916 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.516728 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5377c392-5ccf-4e87-ba8f-69a4fc166088-socket-dir\") pod \"aws-ebs-csi-driver-node-vfb5g\" (UID: \"5377c392-5ccf-4e87-ba8f-69a4fc166088\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfb5g" Jul 08 18:45:41.516916 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.516775 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p9pd\" (UniqueName: \"kubernetes.io/projected/5377c392-5ccf-4e87-ba8f-69a4fc166088-kube-api-access-6p9pd\") pod \"aws-ebs-csi-driver-node-vfb5g\" (UID: \"5377c392-5ccf-4e87-ba8f-69a4fc166088\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfb5g" Jul 08 18:45:41.516916 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.516819 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-log-socket\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.516916 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.516848 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-host-cni-bin\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.516916 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.516879 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-host-cni-netd\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.516916 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.516896 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5377c392-5ccf-4e87-ba8f-69a4fc166088-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vfb5g\" (UID: \"5377c392-5ccf-4e87-ba8f-69a4fc166088\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfb5g" Jul 08 18:45:41.516916 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.516919 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5377c392-5ccf-4e87-ba8f-69a4fc166088-etc-selinux\") pod \"aws-ebs-csi-driver-node-vfb5g\" (UID: \"5377c392-5ccf-4e87-ba8f-69a4fc166088\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfb5g" Jul 08 18:45:41.517357 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.516935 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f79dc66e-2b96-42c4-9ee9-4d036cc63d3a-tmp-dir\") pod \"node-resolver-8hxcn\" (UID: \"f79dc66e-2b96-42c4-9ee9-4d036cc63d3a\") " pod="openshift-dns/node-resolver-8hxcn" Jul 08 18:45:41.517357 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.516951 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-run-systemd\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.517357 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.516916 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 18:45:41.517357 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.517037 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-node-log\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.517357 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.517068 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-multus-cni-dir\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.517357 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.517092 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-os-release\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.517357 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.517115 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-host-var-lib-kubelet\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.517357 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.517157 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-etc-kubernetes\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.517357 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.517181 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-systemd-units\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.517357 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.517208 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-host-slash\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.517357 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.517234 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-var-lib-openvswitch\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.517357 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.517279 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-run-openvswitch\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.517357 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.517341 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-host-run-ovn-kubernetes\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.517842 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.517369 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.517842 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.517409 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c95d4430-b571-42e1-b016-686b9aa91245-env-overrides\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.517842 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.517441 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c95d4430-b571-42e1-b016-686b9aa91245-ovn-node-metrics-cert\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.517842 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.517469 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d8fdda41-866c-40a9-95ff-599de68e4672-konnectivity-ca\") pod \"konnectivity-agent-tfrw6\" (UID: \"d8fdda41-866c-40a9-95ff-599de68e4672\") " pod="kube-system/konnectivity-agent-tfrw6" Jul 08 18:45:41.517842 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.517512 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-multus-socket-dir-parent\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.517842 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.517542 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-host-var-lib-cni-bin\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.517842 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.517568 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-multus-conf-dir\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.517842 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.517595 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm5bl\" (UniqueName: \"kubernetes.io/projected/1e1cf566-6a03-4273-a870-5f78662ca68c-kube-api-access-tm5bl\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.517842 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.517622 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d7c69034-21a4-43da-ab51-09b49ac4a67a-host-slash\") pod \"iptables-alerter-5njd8\" (UID: \"d7c69034-21a4-43da-ab51-09b49ac4a67a\") " pod="openshift-network-operator/iptables-alerter-5njd8" Jul 08 18:45:41.517842 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.517646 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5377c392-5ccf-4e87-ba8f-69a4fc166088-registration-dir\") pod \"aws-ebs-csi-driver-node-vfb5g\" (UID: \"5377c392-5ccf-4e87-ba8f-69a4fc166088\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfb5g" Jul 08 18:45:41.517842 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.517673 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f79dc66e-2b96-42c4-9ee9-4d036cc63d3a-hosts-file\") pod \"node-resolver-8hxcn\" (UID: \"f79dc66e-2b96-42c4-9ee9-4d036cc63d3a\") " pod="openshift-dns/node-resolver-8hxcn" Jul 08 18:45:41.517842 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.517693 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd2d5\" (UniqueName: \"kubernetes.io/projected/c95d4430-b571-42e1-b016-686b9aa91245-kube-api-access-vd2d5\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.517842 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.517716 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e1cf566-6a03-4273-a870-5f78662ca68c-cni-binary-copy\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.518434 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.518274 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-host-run-netns\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.518434 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.518307 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-hostroot\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.537503 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.537482 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sqrhn" Jul 08 18:45:41.546002 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.545982 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sqrhn" Jul 08 18:45:41.606459 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.606426 2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 08 18:45:41.618811 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.618777 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-systemd-units\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.618964 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.618819 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-var-lib-openvswitch\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.618964 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.618845 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-run-openvswitch\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.618964 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.618871 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.618964 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.618898 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c95d4430-b571-42e1-b016-686b9aa91245-env-overrides\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.618964 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.618924 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c95d4430-b571-42e1-b016-686b9aa91245-ovn-node-metrics-cert\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.618964 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.618952 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d8fdda41-866c-40a9-95ff-599de68e4672-konnectivity-ca\") pod \"konnectivity-agent-tfrw6\" (UID: \"d8fdda41-866c-40a9-95ff-599de68e4672\") " pod="kube-system/konnectivity-agent-tfrw6" Jul 08 18:45:41.619276 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.618978 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d7c69034-21a4-43da-ab51-09b49ac4a67a-host-slash\") pod \"iptables-alerter-5njd8\" (UID: \"d7c69034-21a4-43da-ab51-09b49ac4a67a\") " pod="openshift-network-operator/iptables-alerter-5njd8" Jul 08 18:45:41.619276 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.619008 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-host-var-lib-cni-bin\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.619276 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.619032 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-multus-conf-dir\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.619276 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.619057 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm5bl\" (UniqueName: \"kubernetes.io/projected/1e1cf566-6a03-4273-a870-5f78662ca68c-kube-api-access-tm5bl\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.619276 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.619086 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/78c9e9b1-2fc2-47f5-9966-85b349a0f009-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fdwjq\" (UID: \"78c9e9b1-2fc2-47f5-9966-85b349a0f009\") " pod="openshift-multus/multus-additional-cni-plugins-fdwjq" Jul 08 18:45:41.619276 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.619145 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d7c69034-21a4-43da-ab51-09b49ac4a67a-host-slash\") pod \"iptables-alerter-5njd8\" (UID: \"d7c69034-21a4-43da-ab51-09b49ac4a67a\") " pod="openshift-network-operator/iptables-alerter-5njd8" Jul 08 18:45:41.619276 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.619171 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-host-var-lib-cni-bin\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.619611 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.619408 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-multus-conf-dir\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.619611 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.619450 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/02e604f5-7b5c-4aec-a685-e380d4b20a3e-dbus\") pod \"global-pull-secret-syncer-9gjk9\" (UID: \"02e604f5-7b5c-4aec-a685-e380d4b20a3e\") " pod="kube-system/global-pull-secret-syncer-9gjk9" Jul 08 18:45:41.619611 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.619481 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-sys\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.619611 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.619506 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-tmp\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.619611 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.619516 2570 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 08 18:45:41.619611 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.619553 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f79dc66e-2b96-42c4-9ee9-4d036cc63d3a-hosts-file\") pod \"node-resolver-8hxcn\" (UID: \"f79dc66e-2b96-42c4-9ee9-4d036cc63d3a\") " pod="openshift-dns/node-resolver-8hxcn" Jul 08 18:45:41.619611 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.619582 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd2d5\" (UniqueName: \"kubernetes.io/projected/c95d4430-b571-42e1-b016-686b9aa91245-kube-api-access-vd2d5\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.619908 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.619609 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-host-run-netns\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.619908 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.619628 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c95d4430-b571-42e1-b016-686b9aa91245-env-overrides\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.619908 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.619637 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-run\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.619908 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.619666 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a28d7e6-edce-4adf-8200-d499f2b365b0-host\") pod \"node-ca-dqz5x\" (UID: \"7a28d7e6-edce-4adf-8200-d499f2b365b0\") " pod="openshift-image-registry/node-ca-dqz5x" Jul 08 18:45:41.619908 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.619693 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-systemd-units\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.619908 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.619691 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d8fdda41-866c-40a9-95ff-599de68e4672-agent-certs\") pod \"konnectivity-agent-tfrw6\" (UID: \"d8fdda41-866c-40a9-95ff-599de68e4672\") " pod="kube-system/konnectivity-agent-tfrw6" Jul 08 18:45:41.619908 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.619721 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-run-openvswitch\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.619908 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.619737 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/78c9e9b1-2fc2-47f5-9966-85b349a0f009-cni-binary-copy\") pod \"multus-additional-cni-plugins-fdwjq\" (UID: \"78c9e9b1-2fc2-47f5-9966-85b349a0f009\") " pod="openshift-multus/multus-additional-cni-plugins-fdwjq" Jul 08 18:45:41.619908 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.619774 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d8fdda41-866c-40a9-95ff-599de68e4672-konnectivity-ca\") pod \"konnectivity-agent-tfrw6\" (UID: \"d8fdda41-866c-40a9-95ff-599de68e4672\") " pod="kube-system/konnectivity-agent-tfrw6" Jul 08 18:45:41.619908 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.619794 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-var-lib-openvswitch\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.619908 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.619843 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f79dc66e-2b96-42c4-9ee9-4d036cc63d3a-hosts-file\") pod \"node-resolver-8hxcn\" (UID: \"f79dc66e-2b96-42c4-9ee9-4d036cc63d3a\") " pod="openshift-dns/node-resolver-8hxcn" Jul 08 18:45:41.619908 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.619855 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-host-run-netns\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.619908 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.619886 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/02e604f5-7b5c-4aec-a685-e380d4b20a3e-kubelet-config\") pod \"global-pull-secret-syncer-9gjk9\" (UID: \"02e604f5-7b5c-4aec-a685-e380d4b20a3e\") " pod="kube-system/global-pull-secret-syncer-9gjk9" Jul 08 18:45:41.620377 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.619928 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.620377 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.619920 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02e604f5-7b5c-4aec-a685-e380d4b20a3e-original-pull-secret\") pod \"global-pull-secret-syncer-9gjk9\" (UID: \"02e604f5-7b5c-4aec-a685-e380d4b20a3e\") " pod="kube-system/global-pull-secret-syncer-9gjk9" Jul 08 18:45:41.620377 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.619958 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-etc-sysctl-conf\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.620377 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.619991 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt22s\" (UniqueName: \"kubernetes.io/projected/f79dc66e-2b96-42c4-9ee9-4d036cc63d3a-kube-api-access-pt22s\") pod \"node-resolver-8hxcn\" (UID: \"f79dc66e-2b96-42c4-9ee9-4d036cc63d3a\") " pod="openshift-dns/node-resolver-8hxcn" Jul 08 18:45:41.620377 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620020 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-etc-openvswitch\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.620377 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620059 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-system-cni-dir\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.620377 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620084 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1e1cf566-6a03-4273-a870-5f78662ca68c-multus-daemon-config\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.620377 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620103 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-system-cni-dir\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.620377 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620111 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/78c9e9b1-2fc2-47f5-9966-85b349a0f009-cnibin\") pod \"multus-additional-cni-plugins-fdwjq\" (UID: \"78c9e9b1-2fc2-47f5-9966-85b349a0f009\") " pod="openshift-multus/multus-additional-cni-plugins-fdwjq" Jul 08 18:45:41.620377 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620237 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-etc-openvswitch\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.620377 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620268 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-etc-tuned\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.620377 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620301 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c95d4430-b571-42e1-b016-686b9aa91245-ovnkube-config\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.620377 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620325 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-host-var-lib-cni-multus\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.620377 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620348 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p9pd\" (UniqueName: \"kubernetes.io/projected/5377c392-5ccf-4e87-ba8f-69a4fc166088-kube-api-access-6p9pd\") pod \"aws-ebs-csi-driver-node-vfb5g\" (UID: \"5377c392-5ccf-4e87-ba8f-69a4fc166088\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfb5g" Jul 08 18:45:41.620377 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620372 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45fe0fb7-f912-4b72-b298-1dae0ceae220-metrics-certs\") pod \"network-metrics-daemon-mwtbr\" (UID: \"45fe0fb7-f912-4b72-b298-1dae0ceae220\") " pod="openshift-multus/network-metrics-daemon-mwtbr" Jul 08 18:45:41.621064 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620397 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-log-socket\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.621064 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620440 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-host-cni-bin\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.621064 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620462 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-host-cni-netd\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.621064 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620486 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5377c392-5ccf-4e87-ba8f-69a4fc166088-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vfb5g\" (UID: \"5377c392-5ccf-4e87-ba8f-69a4fc166088\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfb5g" Jul 08 18:45:41.621064 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620512 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5377c392-5ccf-4e87-ba8f-69a4fc166088-etc-selinux\") pod \"aws-ebs-csi-driver-node-vfb5g\" (UID: \"5377c392-5ccf-4e87-ba8f-69a4fc166088\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfb5g" Jul 08 18:45:41.621064 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620538 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-node-log\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.621064 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620562 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-multus-cni-dir\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.621064 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620589 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-etc-kubernetes\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.621064 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620604 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-host-cni-bin\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.621064 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620615 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/78c9e9b1-2fc2-47f5-9966-85b349a0f009-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fdwjq\" (UID: \"78c9e9b1-2fc2-47f5-9966-85b349a0f009\") " pod="openshift-multus/multus-additional-cni-plugins-fdwjq" Jul 08 18:45:41.621064 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620632 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1e1cf566-6a03-4273-a870-5f78662ca68c-multus-daemon-config\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.621064 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620649 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prd55\" (UniqueName: \"kubernetes.io/projected/78c9e9b1-2fc2-47f5-9966-85b349a0f009-kube-api-access-prd55\") pod \"multus-additional-cni-plugins-fdwjq\" (UID: \"78c9e9b1-2fc2-47f5-9966-85b349a0f009\") " pod="openshift-multus/multus-additional-cni-plugins-fdwjq" Jul 08 18:45:41.621064 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620678 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-etc-modprobe-d\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.621064 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620667 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-host-var-lib-cni-multus\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.621064 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620700 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-host-cni-netd\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.621064 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620708 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-etc-sysctl-d\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.621064 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620735 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w65rj\" (UniqueName: \"kubernetes.io/projected/45fe0fb7-f912-4b72-b298-1dae0ceae220-kube-api-access-w65rj\") pod \"network-metrics-daemon-mwtbr\" (UID: \"45fe0fb7-f912-4b72-b298-1dae0ceae220\") " pod="openshift-multus/network-metrics-daemon-mwtbr" Jul 08 18:45:41.621886 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620761 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-host-slash\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.621886 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620783 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5377c392-5ccf-4e87-ba8f-69a4fc166088-etc-selinux\") pod \"aws-ebs-csi-driver-node-vfb5g\" (UID: \"5377c392-5ccf-4e87-ba8f-69a4fc166088\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfb5g" Jul 08 18:45:41.621886 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620784 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-host-run-ovn-kubernetes\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.621886 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620829 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-host-run-ovn-kubernetes\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.621886 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620835 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrrk6\" (UniqueName: \"kubernetes.io/projected/7a28d7e6-edce-4adf-8200-d499f2b365b0-kube-api-access-hrrk6\") pod \"node-ca-dqz5x\" (UID: \"7a28d7e6-edce-4adf-8200-d499f2b365b0\") " pod="openshift-image-registry/node-ca-dqz5x" Jul 08 18:45:41.621886 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620867 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-multus-socket-dir-parent\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.621886 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620898 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-multus-cni-dir\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.621886 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620905 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5377c392-5ccf-4e87-ba8f-69a4fc166088-registration-dir\") pod \"aws-ebs-csi-driver-node-vfb5g\" (UID: \"5377c392-5ccf-4e87-ba8f-69a4fc166088\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfb5g" Jul 08 18:45:41.621886 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620913 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-log-socket\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.621886 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620929 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-etc-sysconfig\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.621886 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620945 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5377c392-5ccf-4e87-ba8f-69a4fc166088-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vfb5g\" (UID: \"5377c392-5ccf-4e87-ba8f-69a4fc166088\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfb5g" Jul 08 18:45:41.621886 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620963 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-var-lib-kubelet\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.621886 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621004 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-node-log\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.621886 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.620866 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c95d4430-b571-42e1-b016-686b9aa91245-ovnkube-config\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.621886 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621025 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5377c392-5ccf-4e87-ba8f-69a4fc166088-registration-dir\") pod \"aws-ebs-csi-driver-node-vfb5g\" (UID: \"5377c392-5ccf-4e87-ba8f-69a4fc166088\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfb5g" Jul 08 18:45:41.621886 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621039 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-host-slash\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.621886 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621057 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6dmz\" (UniqueName: \"kubernetes.io/projected/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-kube-api-access-k6dmz\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.622660 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621061 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-multus-socket-dir-parent\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.622660 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621105 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e1cf566-6a03-4273-a870-5f78662ca68c-cni-binary-copy\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.622660 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621131 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-hostroot\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.622660 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621163 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxbtx\" (UniqueName: \"kubernetes.io/projected/f7e42624-526e-4aaf-904f-3541dc59aeda-kube-api-access-zxbtx\") pod \"network-check-target-mmgzx\" (UID: \"f7e42624-526e-4aaf-904f-3541dc59aeda\") " pod="openshift-network-diagnostics/network-check-target-mmgzx" Jul 08 18:45:41.622660 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621190 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7a28d7e6-edce-4adf-8200-d499f2b365b0-serviceca\") pod \"node-ca-dqz5x\" (UID: \"7a28d7e6-edce-4adf-8200-d499f2b365b0\") " pod="openshift-image-registry/node-ca-dqz5x" Jul 08 18:45:41.622660 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621190 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-etc-kubernetes\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.622660 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621230 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-host-kubelet\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.622660 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621241 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-hostroot\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.622660 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621276 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-cnibin\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.622660 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621303 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d7c69034-21a4-43da-ab51-09b49ac4a67a-iptables-alerter-script\") pod \"iptables-alerter-5njd8\" (UID: \"d7c69034-21a4-43da-ab51-09b49ac4a67a\") " pod="openshift-network-operator/iptables-alerter-5njd8" Jul 08 18:45:41.622660 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621364 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-cnibin\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.622660 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621396 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd427\" (UniqueName: \"kubernetes.io/projected/d7c69034-21a4-43da-ab51-09b49ac4a67a-kube-api-access-dd427\") pod \"iptables-alerter-5njd8\" (UID: \"d7c69034-21a4-43da-ab51-09b49ac4a67a\") " pod="openshift-network-operator/iptables-alerter-5njd8" Jul 08 18:45:41.622660 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621425 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5377c392-5ccf-4e87-ba8f-69a4fc166088-device-dir\") pod \"aws-ebs-csi-driver-node-vfb5g\" (UID: \"5377c392-5ccf-4e87-ba8f-69a4fc166088\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfb5g" Jul 08 18:45:41.622660 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621465 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78c9e9b1-2fc2-47f5-9966-85b349a0f009-system-cni-dir\") pod \"multus-additional-cni-plugins-fdwjq\" (UID: \"78c9e9b1-2fc2-47f5-9966-85b349a0f009\") " pod="openshift-multus/multus-additional-cni-plugins-fdwjq" Jul 08 18:45:41.622660 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621485 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-host-kubelet\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.622660 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621492 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-host\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.622660 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621518 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-run-ovn\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.622660 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621543 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-host-run-k8s-cni-cncf-io\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.623530 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621569 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-host-run-multus-certs\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.623530 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621596 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5377c392-5ccf-4e87-ba8f-69a4fc166088-sys-fs\") pod \"aws-ebs-csi-driver-node-vfb5g\" (UID: \"5377c392-5ccf-4e87-ba8f-69a4fc166088\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfb5g" Jul 08 18:45:41.623530 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621613 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e1cf566-6a03-4273-a870-5f78662ca68c-cni-binary-copy\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.623530 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621654 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-lib-modules\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.623530 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621664 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-host-run-multus-certs\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.623530 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621682 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-host-run-netns\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.623530 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621708 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c95d4430-b571-42e1-b016-686b9aa91245-ovnkube-script-lib\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.623530 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621715 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5377c392-5ccf-4e87-ba8f-69a4fc166088-device-dir\") pod \"aws-ebs-csi-driver-node-vfb5g\" (UID: \"5377c392-5ccf-4e87-ba8f-69a4fc166088\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfb5g" Jul 08 18:45:41.623530 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621734 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5377c392-5ccf-4e87-ba8f-69a4fc166088-socket-dir\") pod \"aws-ebs-csi-driver-node-vfb5g\" (UID: \"5377c392-5ccf-4e87-ba8f-69a4fc166088\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfb5g" Jul 08 18:45:41.623530 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621743 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-host-run-k8s-cni-cncf-io\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.623530 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621756 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5377c392-5ccf-4e87-ba8f-69a4fc166088-sys-fs\") pod \"aws-ebs-csi-driver-node-vfb5g\" (UID: \"5377c392-5ccf-4e87-ba8f-69a4fc166088\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfb5g" Jul 08 18:45:41.623530 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621805 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/78c9e9b1-2fc2-47f5-9966-85b349a0f009-os-release\") pod \"multus-additional-cni-plugins-fdwjq\" (UID: \"78c9e9b1-2fc2-47f5-9966-85b349a0f009\") " pod="openshift-multus/multus-additional-cni-plugins-fdwjq" Jul 08 18:45:41.623530 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621832 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/78c9e9b1-2fc2-47f5-9966-85b349a0f009-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fdwjq\" (UID: \"78c9e9b1-2fc2-47f5-9966-85b349a0f009\") " pod="openshift-multus/multus-additional-cni-plugins-fdwjq" Jul 08 18:45:41.623530 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621838 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-host-run-netns\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.623530 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621856 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-etc-systemd\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.623530 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621870 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d7c69034-21a4-43da-ab51-09b49ac4a67a-iptables-alerter-script\") pod \"iptables-alerter-5njd8\" (UID: \"d7c69034-21a4-43da-ab51-09b49ac4a67a\") " pod="openshift-network-operator/iptables-alerter-5njd8" Jul 08 18:45:41.623530 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621893 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f79dc66e-2b96-42c4-9ee9-4d036cc63d3a-tmp-dir\") pod \"node-resolver-8hxcn\" (UID: \"f79dc66e-2b96-42c4-9ee9-4d036cc63d3a\") " pod="openshift-dns/node-resolver-8hxcn" Jul 08 18:45:41.624028 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621919 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-run-systemd\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.624028 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621944 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-os-release\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.624028 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621969 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-host-var-lib-kubelet\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.624028 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621989 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5377c392-5ccf-4e87-ba8f-69a4fc166088-socket-dir\") pod \"aws-ebs-csi-driver-node-vfb5g\" (UID: \"5377c392-5ccf-4e87-ba8f-69a4fc166088\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfb5g" Jul 08 18:45:41.624028 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.621997 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-etc-kubernetes\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.624028 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.622076 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-host-var-lib-kubelet\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.624028 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.622121 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-run-ovn\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.624028 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.622168 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c95d4430-b571-42e1-b016-686b9aa91245-run-systemd\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.624028 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.622223 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1e1cf566-6a03-4273-a870-5f78662ca68c-os-release\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.624028 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.622311 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f79dc66e-2b96-42c4-9ee9-4d036cc63d3a-tmp-dir\") pod \"node-resolver-8hxcn\" (UID: \"f79dc66e-2b96-42c4-9ee9-4d036cc63d3a\") " pod="openshift-dns/node-resolver-8hxcn" Jul 08 18:45:41.624028 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.622409 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c95d4430-b571-42e1-b016-686b9aa91245-ovnkube-script-lib\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.624028 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.623158 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c95d4430-b571-42e1-b016-686b9aa91245-ovn-node-metrics-cert\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.624028 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.623893 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d8fdda41-866c-40a9-95ff-599de68e4672-agent-certs\") pod \"konnectivity-agent-tfrw6\" (UID: \"d8fdda41-866c-40a9-95ff-599de68e4672\") " pod="kube-system/konnectivity-agent-tfrw6" Jul 08 18:45:41.628215 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.628189 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm5bl\" (UniqueName: \"kubernetes.io/projected/1e1cf566-6a03-4273-a870-5f78662ca68c-kube-api-access-tm5bl\") pod \"multus-c2vbf\" (UID: \"1e1cf566-6a03-4273-a870-5f78662ca68c\") " pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.629434 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.629405 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd2d5\" (UniqueName: \"kubernetes.io/projected/c95d4430-b571-42e1-b016-686b9aa91245-kube-api-access-vd2d5\") pod \"ovnkube-node-wqmlf\" (UID: \"c95d4430-b571-42e1-b016-686b9aa91245\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.630161 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.630121 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt22s\" (UniqueName: \"kubernetes.io/projected/f79dc66e-2b96-42c4-9ee9-4d036cc63d3a-kube-api-access-pt22s\") pod \"node-resolver-8hxcn\" (UID: \"f79dc66e-2b96-42c4-9ee9-4d036cc63d3a\") " pod="openshift-dns/node-resolver-8hxcn" Jul 08 18:45:41.631080 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.631056 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p9pd\" (UniqueName: \"kubernetes.io/projected/5377c392-5ccf-4e87-ba8f-69a4fc166088-kube-api-access-6p9pd\") pod \"aws-ebs-csi-driver-node-vfb5g\" (UID: \"5377c392-5ccf-4e87-ba8f-69a4fc166088\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfb5g" Jul 08 18:45:41.632366 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:41.632346 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 18:45:41.632366 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:41.632367 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 18:45:41.632516 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:41.632379 2570 projected.go:196] Error preparing data for projected volume kube-api-access-zxbtx for pod openshift-network-diagnostics/network-check-target-mmgzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:45:41.632516 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:41.632454 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7e42624-526e-4aaf-904f-3541dc59aeda-kube-api-access-zxbtx podName:f7e42624-526e-4aaf-904f-3541dc59aeda nodeName:}" failed. No retries permitted until 2026-07-08 18:45:42.132427502 +0000 UTC m=+2.979702146 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zxbtx" (UniqueName: "kubernetes.io/projected/f7e42624-526e-4aaf-904f-3541dc59aeda-kube-api-access-zxbtx") pod "network-check-target-mmgzx" (UID: "f7e42624-526e-4aaf-904f-3541dc59aeda") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:45:41.634636 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.634608 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd427\" (UniqueName: \"kubernetes.io/projected/d7c69034-21a4-43da-ab51-09b49ac4a67a-kube-api-access-dd427\") pod \"iptables-alerter-5njd8\" (UID: \"d7c69034-21a4-43da-ab51-09b49ac4a67a\") " pod="openshift-network-operator/iptables-alerter-5njd8" Jul 08 18:45:41.723135 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.723092 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/78c9e9b1-2fc2-47f5-9966-85b349a0f009-cnibin\") pod \"multus-additional-cni-plugins-fdwjq\" (UID: \"78c9e9b1-2fc2-47f5-9966-85b349a0f009\") " pod="openshift-multus/multus-additional-cni-plugins-fdwjq" Jul 08 18:45:41.723135 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.723136 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-etc-tuned\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.723421 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.723164 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45fe0fb7-f912-4b72-b298-1dae0ceae220-metrics-certs\") pod \"network-metrics-daemon-mwtbr\" (UID: \"45fe0fb7-f912-4b72-b298-1dae0ceae220\") " pod="openshift-multus/network-metrics-daemon-mwtbr" Jul 08 18:45:41.723421 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.723183 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/78c9e9b1-2fc2-47f5-9966-85b349a0f009-cnibin\") pod \"multus-additional-cni-plugins-fdwjq\" (UID: \"78c9e9b1-2fc2-47f5-9966-85b349a0f009\") " pod="openshift-multus/multus-additional-cni-plugins-fdwjq" Jul 08 18:45:41.723421 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.723195 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/78c9e9b1-2fc2-47f5-9966-85b349a0f009-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fdwjq\" (UID: \"78c9e9b1-2fc2-47f5-9966-85b349a0f009\") " pod="openshift-multus/multus-additional-cni-plugins-fdwjq" Jul 08 18:45:41.723421 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.723237 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prd55\" (UniqueName: \"kubernetes.io/projected/78c9e9b1-2fc2-47f5-9966-85b349a0f009-kube-api-access-prd55\") pod \"multus-additional-cni-plugins-fdwjq\" (UID: \"78c9e9b1-2fc2-47f5-9966-85b349a0f009\") " pod="openshift-multus/multus-additional-cni-plugins-fdwjq" Jul 08 18:45:41.723421 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.723279 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-etc-modprobe-d\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.723421 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.723302 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-etc-sysctl-d\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.723421 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.723324 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w65rj\" (UniqueName: \"kubernetes.io/projected/45fe0fb7-f912-4b72-b298-1dae0ceae220-kube-api-access-w65rj\") pod \"network-metrics-daemon-mwtbr\" (UID: \"45fe0fb7-f912-4b72-b298-1dae0ceae220\") " pod="openshift-multus/network-metrics-daemon-mwtbr" Jul 08 18:45:41.723421 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:41.723326 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:45:41.723421 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.723411 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/78c9e9b1-2fc2-47f5-9966-85b349a0f009-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fdwjq\" (UID: \"78c9e9b1-2fc2-47f5-9966-85b349a0f009\") " pod="openshift-multus/multus-additional-cni-plugins-fdwjq" Jul 08 18:45:41.723421 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:41.723416 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45fe0fb7-f912-4b72-b298-1dae0ceae220-metrics-certs podName:45fe0fb7-f912-4b72-b298-1dae0ceae220 nodeName:}" failed. No retries permitted until 2026-07-08 18:45:42.223397699 +0000 UTC m=+3.070672357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45fe0fb7-f912-4b72-b298-1dae0ceae220-metrics-certs") pod "network-metrics-daemon-mwtbr" (UID: "45fe0fb7-f912-4b72-b298-1dae0ceae220") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:45:41.723900 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.723446 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-etc-sysctl-d\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.723900 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.723459 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-etc-modprobe-d\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.723900 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.723488 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrrk6\" (UniqueName: \"kubernetes.io/projected/7a28d7e6-edce-4adf-8200-d499f2b365b0-kube-api-access-hrrk6\") pod \"node-ca-dqz5x\" (UID: \"7a28d7e6-edce-4adf-8200-d499f2b365b0\") " pod="openshift-image-registry/node-ca-dqz5x" Jul 08 18:45:41.723900 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.723515 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-etc-sysconfig\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.723900 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.723564 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-etc-sysconfig\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.723900 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.723636 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-var-lib-kubelet\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.723900 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.723672 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6dmz\" (UniqueName: \"kubernetes.io/projected/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-kube-api-access-k6dmz\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.723900 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.723713 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7a28d7e6-edce-4adf-8200-d499f2b365b0-serviceca\") pod \"node-ca-dqz5x\" (UID: \"7a28d7e6-edce-4adf-8200-d499f2b365b0\") " pod="openshift-image-registry/node-ca-dqz5x" Jul 08 18:45:41.723900 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.723734 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-var-lib-kubelet\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.723900 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.723743 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78c9e9b1-2fc2-47f5-9966-85b349a0f009-system-cni-dir\") pod \"multus-additional-cni-plugins-fdwjq\" (UID: \"78c9e9b1-2fc2-47f5-9966-85b349a0f009\") " pod="openshift-multus/multus-additional-cni-plugins-fdwjq" Jul 08 18:45:41.723900 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.723782 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-host\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.723900 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.723794 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78c9e9b1-2fc2-47f5-9966-85b349a0f009-system-cni-dir\") pod \"multus-additional-cni-plugins-fdwjq\" (UID: \"78c9e9b1-2fc2-47f5-9966-85b349a0f009\") " pod="openshift-multus/multus-additional-cni-plugins-fdwjq" Jul 08 18:45:41.723900 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.723807 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-lib-modules\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.723900 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.723836 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/78c9e9b1-2fc2-47f5-9966-85b349a0f009-os-release\") pod \"multus-additional-cni-plugins-fdwjq\" (UID: \"78c9e9b1-2fc2-47f5-9966-85b349a0f009\") " pod="openshift-multus/multus-additional-cni-plugins-fdwjq" Jul 08 18:45:41.723900 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.723853 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-host\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.723900 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.723861 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/78c9e9b1-2fc2-47f5-9966-85b349a0f009-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fdwjq\" (UID: \"78c9e9b1-2fc2-47f5-9966-85b349a0f009\") " pod="openshift-multus/multus-additional-cni-plugins-fdwjq" Jul 08 18:45:41.723900 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.723899 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-etc-systemd\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.724700 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.723931 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-etc-kubernetes\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.724700 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.723971 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/78c9e9b1-2fc2-47f5-9966-85b349a0f009-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fdwjq\" (UID: \"78c9e9b1-2fc2-47f5-9966-85b349a0f009\") " pod="openshift-multus/multus-additional-cni-plugins-fdwjq" Jul 08 18:45:41.724700 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.723977 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-etc-systemd\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.724700 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.724001 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-lib-modules\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.724700 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.724010 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/02e604f5-7b5c-4aec-a685-e380d4b20a3e-dbus\") pod \"global-pull-secret-syncer-9gjk9\" (UID: \"02e604f5-7b5c-4aec-a685-e380d4b20a3e\") " pod="kube-system/global-pull-secret-syncer-9gjk9" Jul 08 18:45:41.724700 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.724034 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-sys\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.724700 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.724063 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/78c9e9b1-2fc2-47f5-9966-85b349a0f009-os-release\") pod \"multus-additional-cni-plugins-fdwjq\" (UID: \"78c9e9b1-2fc2-47f5-9966-85b349a0f009\") " pod="openshift-multus/multus-additional-cni-plugins-fdwjq" Jul 08 18:45:41.724700 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.724070 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-etc-kubernetes\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.724700 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.724056 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-tmp\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.724700 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.724117 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-run\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.724700 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.724143 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a28d7e6-edce-4adf-8200-d499f2b365b0-host\") pod \"node-ca-dqz5x\" (UID: \"7a28d7e6-edce-4adf-8200-d499f2b365b0\") " pod="openshift-image-registry/node-ca-dqz5x" Jul 08 18:45:41.724700 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.724172 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/78c9e9b1-2fc2-47f5-9966-85b349a0f009-cni-binary-copy\") pod \"multus-additional-cni-plugins-fdwjq\" (UID: \"78c9e9b1-2fc2-47f5-9966-85b349a0f009\") " pod="openshift-multus/multus-additional-cni-plugins-fdwjq" Jul 08 18:45:41.724700 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.724201 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/02e604f5-7b5c-4aec-a685-e380d4b20a3e-kubelet-config\") pod \"global-pull-secret-syncer-9gjk9\" (UID: \"02e604f5-7b5c-4aec-a685-e380d4b20a3e\") " pod="kube-system/global-pull-secret-syncer-9gjk9" Jul 08 18:45:41.724700 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.724221 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-run\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.724700 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.724229 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02e604f5-7b5c-4aec-a685-e380d4b20a3e-original-pull-secret\") pod \"global-pull-secret-syncer-9gjk9\" (UID: \"02e604f5-7b5c-4aec-a685-e380d4b20a3e\") " pod="kube-system/global-pull-secret-syncer-9gjk9" Jul 08 18:45:41.724700 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.724174 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/02e604f5-7b5c-4aec-a685-e380d4b20a3e-dbus\") pod \"global-pull-secret-syncer-9gjk9\" (UID: \"02e604f5-7b5c-4aec-a685-e380d4b20a3e\") " pod="kube-system/global-pull-secret-syncer-9gjk9" Jul 08 18:45:41.724700 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.724239 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7a28d7e6-edce-4adf-8200-d499f2b365b0-serviceca\") pod \"node-ca-dqz5x\" (UID: \"7a28d7e6-edce-4adf-8200-d499f2b365b0\") " pod="openshift-image-registry/node-ca-dqz5x" Jul 08 18:45:41.724700 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.724273 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-etc-sysctl-conf\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.725465 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.724366 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/02e604f5-7b5c-4aec-a685-e380d4b20a3e-kubelet-config\") pod \"global-pull-secret-syncer-9gjk9\" (UID: \"02e604f5-7b5c-4aec-a685-e380d4b20a3e\") " pod="kube-system/global-pull-secret-syncer-9gjk9" Jul 08 18:45:41.725465 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.724391 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-etc-sysctl-conf\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.725465 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.724415 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a28d7e6-edce-4adf-8200-d499f2b365b0-host\") pod \"node-ca-dqz5x\" (UID: \"7a28d7e6-edce-4adf-8200-d499f2b365b0\") " pod="openshift-image-registry/node-ca-dqz5x" Jul 08 18:45:41.725465 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.724463 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/78c9e9b1-2fc2-47f5-9966-85b349a0f009-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fdwjq\" (UID: \"78c9e9b1-2fc2-47f5-9966-85b349a0f009\") " pod="openshift-multus/multus-additional-cni-plugins-fdwjq" Jul 08 18:45:41.725465 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:41.724481 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 18:45:41.725465 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:41.724562 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02e604f5-7b5c-4aec-a685-e380d4b20a3e-original-pull-secret podName:02e604f5-7b5c-4aec-a685-e380d4b20a3e nodeName:}" failed. No retries permitted until 2026-07-08 18:45:42.224545536 +0000 UTC m=+3.071820179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/02e604f5-7b5c-4aec-a685-e380d4b20a3e-original-pull-secret") pod "global-pull-secret-syncer-9gjk9" (UID: "02e604f5-7b5c-4aec-a685-e380d4b20a3e") : object "kube-system"/"original-pull-secret" not registered Jul 08 18:45:41.725465 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.724626 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/78c9e9b1-2fc2-47f5-9966-85b349a0f009-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fdwjq\" (UID: \"78c9e9b1-2fc2-47f5-9966-85b349a0f009\") " pod="openshift-multus/multus-additional-cni-plugins-fdwjq" Jul 08 18:45:41.725465 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.724605 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-sys\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.725465 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.724680 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/78c9e9b1-2fc2-47f5-9966-85b349a0f009-cni-binary-copy\") pod \"multus-additional-cni-plugins-fdwjq\" (UID: \"78c9e9b1-2fc2-47f5-9966-85b349a0f009\") " pod="openshift-multus/multus-additional-cni-plugins-fdwjq" Jul 08 18:45:41.726007 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.725988 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-etc-tuned\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.726344 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.726322 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-tmp\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.734662 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.734594 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrrk6\" (UniqueName: \"kubernetes.io/projected/7a28d7e6-edce-4adf-8200-d499f2b365b0-kube-api-access-hrrk6\") pod \"node-ca-dqz5x\" (UID: \"7a28d7e6-edce-4adf-8200-d499f2b365b0\") " pod="openshift-image-registry/node-ca-dqz5x" Jul 08 18:45:41.734936 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.734913 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6dmz\" (UniqueName: \"kubernetes.io/projected/1b90c2de-d63a-429c-8a95-7de30dc0b0ed-kube-api-access-k6dmz\") pod \"tuned-2jcvm\" (UID: \"1b90c2de-d63a-429c-8a95-7de30dc0b0ed\") " pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.735291 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.735248 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w65rj\" (UniqueName: \"kubernetes.io/projected/45fe0fb7-f912-4b72-b298-1dae0ceae220-kube-api-access-w65rj\") pod \"network-metrics-daemon-mwtbr\" (UID: \"45fe0fb7-f912-4b72-b298-1dae0ceae220\") " pod="openshift-multus/network-metrics-daemon-mwtbr" Jul 08 18:45:41.736606 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.736591 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prd55\" (UniqueName: \"kubernetes.io/projected/78c9e9b1-2fc2-47f5-9966-85b349a0f009-kube-api-access-prd55\") pod \"multus-additional-cni-plugins-fdwjq\" (UID: \"78c9e9b1-2fc2-47f5-9966-85b349a0f009\") " pod="openshift-multus/multus-additional-cni-plugins-fdwjq" Jul 08 18:45:41.808988 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.808954 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8hxcn" Jul 08 18:45:41.820461 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.820434 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5njd8" Jul 08 18:45:41.839455 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.839427 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:45:41.844314 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.844283 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tfrw6" Jul 08 18:45:41.851020 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.850995 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfb5g" Jul 08 18:45:41.858945 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.858923 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c2vbf" Jul 08 18:45:41.864743 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.864725 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" Jul 08 18:45:41.872336 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.872317 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fdwjq" Jul 08 18:45:41.877807 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:41.877789 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dqz5x" Jul 08 18:45:42.228354 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:42.228243 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxbtx\" (UniqueName: \"kubernetes.io/projected/f7e42624-526e-4aaf-904f-3541dc59aeda-kube-api-access-zxbtx\") pod \"network-check-target-mmgzx\" (UID: \"f7e42624-526e-4aaf-904f-3541dc59aeda\") " pod="openshift-network-diagnostics/network-check-target-mmgzx" Jul 08 18:45:42.228354 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:42.228342 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02e604f5-7b5c-4aec-a685-e380d4b20a3e-original-pull-secret\") pod \"global-pull-secret-syncer-9gjk9\" (UID: \"02e604f5-7b5c-4aec-a685-e380d4b20a3e\") " pod="kube-system/global-pull-secret-syncer-9gjk9" Jul 08 18:45:42.228585 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:42.228380 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45fe0fb7-f912-4b72-b298-1dae0ceae220-metrics-certs\") pod \"network-metrics-daemon-mwtbr\" (UID: \"45fe0fb7-f912-4b72-b298-1dae0ceae220\") " pod="openshift-multus/network-metrics-daemon-mwtbr" Jul 08 18:45:42.228585 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:42.228435 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 18:45:42.228585 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:42.228462 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 18:45:42.228585 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:42.228474 2570 projected.go:196] Error preparing data for projected volume kube-api-access-zxbtx for pod openshift-network-diagnostics/network-check-target-mmgzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:45:42.228585 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:42.228474 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:45:42.228585 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:42.228474 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 18:45:42.228585 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:42.228534 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7e42624-526e-4aaf-904f-3541dc59aeda-kube-api-access-zxbtx podName:f7e42624-526e-4aaf-904f-3541dc59aeda nodeName:}" failed. No retries permitted until 2026-07-08 18:45:43.228516037 +0000 UTC m=+4.075790701 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zxbtx" (UniqueName: "kubernetes.io/projected/f7e42624-526e-4aaf-904f-3541dc59aeda-kube-api-access-zxbtx") pod "network-check-target-mmgzx" (UID: "f7e42624-526e-4aaf-904f-3541dc59aeda") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:45:42.228585 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:42.228558 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45fe0fb7-f912-4b72-b298-1dae0ceae220-metrics-certs podName:45fe0fb7-f912-4b72-b298-1dae0ceae220 nodeName:}" failed. No retries permitted until 2026-07-08 18:45:43.22854651 +0000 UTC m=+4.075821157 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45fe0fb7-f912-4b72-b298-1dae0ceae220-metrics-certs") pod "network-metrics-daemon-mwtbr" (UID: "45fe0fb7-f912-4b72-b298-1dae0ceae220") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:45:42.228585 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:42.228575 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02e604f5-7b5c-4aec-a685-e380d4b20a3e-original-pull-secret podName:02e604f5-7b5c-4aec-a685-e380d4b20a3e nodeName:}" failed. No retries permitted until 2026-07-08 18:45:43.228566558 +0000 UTC m=+4.075841203 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/02e604f5-7b5c-4aec-a685-e380d4b20a3e-original-pull-secret") pod "global-pull-secret-syncer-9gjk9" (UID: "02e604f5-7b5c-4aec-a685-e380d4b20a3e") : object "kube-system"/"original-pull-secret" not registered Jul 08 18:45:42.548857 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:42.548745 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 18:40:41 +0000 UTC" deadline="2028-03-28 19:41:22.85949185 +0000 UTC" Jul 08 18:45:42.548857 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:42.548811 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15096h55m40.310686773s" Jul 08 18:45:42.651849 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:42.651817 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5377c392_5ccf_4e87_ba8f_69a4fc166088.slice/crio-080083a07638eb6b8cd272d73d02dc90b1cadf33936cab37b3e271edaaad768a WatchSource:0}: Error finding container 080083a07638eb6b8cd272d73d02dc90b1cadf33936cab37b3e271edaaad768a: Status 404 returned error can't find the container with id 080083a07638eb6b8cd272d73d02dc90b1cadf33936cab37b3e271edaaad768a Jul 08 18:45:42.654908 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:42.654877 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7c69034_21a4_43da_ab51_09b49ac4a67a.slice/crio-abb5d0aad37ba114844b8d87d1aa44dc98dce8a094a22b17d576e78143c874b1 WatchSource:0}: Error finding container abb5d0aad37ba114844b8d87d1aa44dc98dce8a094a22b17d576e78143c874b1: Status 404 returned error can't find the container with id abb5d0aad37ba114844b8d87d1aa44dc98dce8a094a22b17d576e78143c874b1 Jul 08 18:45:42.657418 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:42.657394 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c9e9b1_2fc2_47f5_9966_85b349a0f009.slice/crio-dacb7b9d80c7e25575f02418e0483ae5b869728eb95abf4c93ee0a0b4bc43324 WatchSource:0}: Error finding container dacb7b9d80c7e25575f02418e0483ae5b869728eb95abf4c93ee0a0b4bc43324: Status 404 returned error can't find the container with id dacb7b9d80c7e25575f02418e0483ae5b869728eb95abf4c93ee0a0b4bc43324 Jul 08 18:45:42.658237 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:42.658220 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b90c2de_d63a_429c_8a95_7de30dc0b0ed.slice/crio-0af022ac822320ea42b8021309608b4307e72738a2c11abdc39376fe03b9a66d WatchSource:0}: Error finding container 0af022ac822320ea42b8021309608b4307e72738a2c11abdc39376fe03b9a66d: Status 404 returned error can't find the container with id 0af022ac822320ea42b8021309608b4307e72738a2c11abdc39376fe03b9a66d Jul 08 18:45:42.659503 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:42.659480 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e1cf566_6a03_4273_a870_5f78662ca68c.slice/crio-9ca5ebe6196539447766599f174bf87605b4fbe840faafc4237ebd9b5faf5adf WatchSource:0}: Error finding container 9ca5ebe6196539447766599f174bf87605b4fbe840faafc4237ebd9b5faf5adf: Status 404 returned error can't find the container with id 9ca5ebe6196539447766599f174bf87605b4fbe840faafc4237ebd9b5faf5adf Jul 08 18:45:42.660712 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:42.660413 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a28d7e6_edce_4adf_8200_d499f2b365b0.slice/crio-e2e53bf3fd7c61ad6e1f6d4ff879134021eeaccdffaf8b71facbb307093e7016 WatchSource:0}: Error finding container e2e53bf3fd7c61ad6e1f6d4ff879134021eeaccdffaf8b71facbb307093e7016: Status 404 returned error can't find the container with id e2e53bf3fd7c61ad6e1f6d4ff879134021eeaccdffaf8b71facbb307093e7016 Jul 08 18:45:42.661108 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:42.660947 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf79dc66e_2b96_42c4_9ee9_4d036cc63d3a.slice/crio-e6cd9a4c71642cf740bc92c4f994a888304027b6935864290e739cf1d6aac964 WatchSource:0}: Error finding container e6cd9a4c71642cf740bc92c4f994a888304027b6935864290e739cf1d6aac964: Status 404 returned error can't find the container with id e6cd9a4c71642cf740bc92c4f994a888304027b6935864290e739cf1d6aac964 Jul 08 18:45:42.661945 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:45:42.661842 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8fdda41_866c_40a9_95ff_599de68e4672.slice/crio-7d4b50831aa45bbac8fecbd5922b5888e17318423d7216eec2a7def24d8a66b1 WatchSource:0}: Error finding container 7d4b50831aa45bbac8fecbd5922b5888e17318423d7216eec2a7def24d8a66b1: Status 404 returned error can't find the container with id 7d4b50831aa45bbac8fecbd5922b5888e17318423d7216eec2a7def24d8a66b1 Jul 08 18:45:42.683524 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:42.683499 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9gjk9" Jul 08 18:45:42.683609 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:42.683499 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwtbr" Jul 08 18:45:42.683664 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:42.683509 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmgzx" Jul 08 18:45:42.683664 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:42.683649 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9gjk9" podUID="02e604f5-7b5c-4aec-a685-e380d4b20a3e" Jul 08 18:45:42.683773 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:42.683705 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwtbr" podUID="45fe0fb7-f912-4b72-b298-1dae0ceae220" Jul 08 18:45:42.683773 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:42.683756 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mmgzx" podUID="f7e42624-526e-4aaf-904f-3541dc59aeda" Jul 08 18:45:42.692214 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:42.692089 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5njd8" event={"ID":"d7c69034-21a4-43da-ab51-09b49ac4a67a","Type":"ContainerStarted","Data":"abb5d0aad37ba114844b8d87d1aa44dc98dce8a094a22b17d576e78143c874b1"} Jul 08 18:45:42.697576 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:42.697551 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" event={"ID":"c95d4430-b571-42e1-b016-686b9aa91245","Type":"ContainerStarted","Data":"3175ca90f95c6ad8007bafd6b26a5fd0d595d8973804de78d43dc401a8d20668"} Jul 08 18:45:42.701037 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:42.701010 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tfrw6" event={"ID":"d8fdda41-866c-40a9-95ff-599de68e4672","Type":"ContainerStarted","Data":"7d4b50831aa45bbac8fecbd5922b5888e17318423d7216eec2a7def24d8a66b1"} Jul 08 18:45:42.701983 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:42.701961 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dqz5x" event={"ID":"7a28d7e6-edce-4adf-8200-d499f2b365b0","Type":"ContainerStarted","Data":"e2e53bf3fd7c61ad6e1f6d4ff879134021eeaccdffaf8b71facbb307093e7016"} Jul 08 18:45:42.702959 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:42.702940 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c2vbf" event={"ID":"1e1cf566-6a03-4273-a870-5f78662ca68c","Type":"ContainerStarted","Data":"9ca5ebe6196539447766599f174bf87605b4fbe840faafc4237ebd9b5faf5adf"} Jul 08 18:45:42.704271 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:42.704204 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fdwjq" event={"ID":"78c9e9b1-2fc2-47f5-9966-85b349a0f009","Type":"ContainerStarted","Data":"dacb7b9d80c7e25575f02418e0483ae5b869728eb95abf4c93ee0a0b4bc43324"} Jul 08 18:45:42.705301 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:42.705282 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfb5g" event={"ID":"5377c392-5ccf-4e87-ba8f-69a4fc166088","Type":"ContainerStarted","Data":"080083a07638eb6b8cd272d73d02dc90b1cadf33936cab37b3e271edaaad768a"} Jul 08 18:45:42.706291 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:42.706270 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" event={"ID":"1b90c2de-d63a-429c-8a95-7de30dc0b0ed","Type":"ContainerStarted","Data":"0af022ac822320ea42b8021309608b4307e72738a2c11abdc39376fe03b9a66d"} Jul 08 18:45:42.707224 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:42.707206 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8hxcn" event={"ID":"f79dc66e-2b96-42c4-9ee9-4d036cc63d3a","Type":"ContainerStarted","Data":"e6cd9a4c71642cf740bc92c4f994a888304027b6935864290e739cf1d6aac964"} Jul 08 18:45:43.235619 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:43.235531 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxbtx\" (UniqueName: \"kubernetes.io/projected/f7e42624-526e-4aaf-904f-3541dc59aeda-kube-api-access-zxbtx\") pod \"network-check-target-mmgzx\" (UID: \"f7e42624-526e-4aaf-904f-3541dc59aeda\") " pod="openshift-network-diagnostics/network-check-target-mmgzx" Jul 08 18:45:43.235619 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:43.235619 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02e604f5-7b5c-4aec-a685-e380d4b20a3e-original-pull-secret\") pod \"global-pull-secret-syncer-9gjk9\" (UID: \"02e604f5-7b5c-4aec-a685-e380d4b20a3e\") " pod="kube-system/global-pull-secret-syncer-9gjk9" Jul 08 18:45:43.235827 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:43.235651 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45fe0fb7-f912-4b72-b298-1dae0ceae220-metrics-certs\") pod \"network-metrics-daemon-mwtbr\" (UID: \"45fe0fb7-f912-4b72-b298-1dae0ceae220\") " pod="openshift-multus/network-metrics-daemon-mwtbr" Jul 08 18:45:43.235827 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:43.235770 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:45:43.235943 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:43.235840 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45fe0fb7-f912-4b72-b298-1dae0ceae220-metrics-certs podName:45fe0fb7-f912-4b72-b298-1dae0ceae220 nodeName:}" failed. No retries permitted until 2026-07-08 18:45:45.235818124 +0000 UTC m=+6.083092771 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45fe0fb7-f912-4b72-b298-1dae0ceae220-metrics-certs") pod "network-metrics-daemon-mwtbr" (UID: "45fe0fb7-f912-4b72-b298-1dae0ceae220") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:45:43.236005 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:43.235944 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 18:45:43.236005 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:43.235971 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 18:45:43.236005 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:43.235986 2570 projected.go:196] Error preparing data for projected volume kube-api-access-zxbtx for pod openshift-network-diagnostics/network-check-target-mmgzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:45:43.236145 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:43.236046 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7e42624-526e-4aaf-904f-3541dc59aeda-kube-api-access-zxbtx podName:f7e42624-526e-4aaf-904f-3541dc59aeda nodeName:}" failed. No retries permitted until 2026-07-08 18:45:45.236024237 +0000 UTC m=+6.083298887 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zxbtx" (UniqueName: "kubernetes.io/projected/f7e42624-526e-4aaf-904f-3541dc59aeda-kube-api-access-zxbtx") pod "network-check-target-mmgzx" (UID: "f7e42624-526e-4aaf-904f-3541dc59aeda") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:45:43.236145 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:43.236120 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 18:45:43.236243 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:43.236159 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02e604f5-7b5c-4aec-a685-e380d4b20a3e-original-pull-secret podName:02e604f5-7b5c-4aec-a685-e380d4b20a3e nodeName:}" failed. No retries permitted until 2026-07-08 18:45:45.236147666 +0000 UTC m=+6.083422317 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/02e604f5-7b5c-4aec-a685-e380d4b20a3e-original-pull-secret") pod "global-pull-secret-syncer-9gjk9" (UID: "02e604f5-7b5c-4aec-a685-e380d4b20a3e") : object "kube-system"/"original-pull-secret" not registered Jul 08 18:45:43.550004 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:43.549898 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 18:40:41 +0000 UTC" deadline="2028-01-23 10:28:23.56333075 +0000 UTC" Jul 08 18:45:43.550004 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:43.549965 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13527h42m40.013370602s" Jul 08 18:45:43.721512 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:43.721466 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-48.ec2.internal" event={"ID":"258c89e0ffadf0a34366b9c773cb3f43","Type":"ContainerStarted","Data":"7c08a590effddd81df774e89c4cd55fff12538aaee183841bcc8f0d9cdb1499e"} Jul 08 18:45:43.734559 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:43.734526 2570 generic.go:356] "Generic (PLEG): container finished" podID="3d7ba020ad0d6e8fc7260c108455cdc7" containerID="25f5e54e63b49f725f6b3c00925860f9ca21c1c29c81fd4ee66fb698d68a991c" exitCode=0 Jul 08 18:45:43.734724 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:43.734572 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-48.ec2.internal" event={"ID":"3d7ba020ad0d6e8fc7260c108455cdc7","Type":"ContainerDied","Data":"25f5e54e63b49f725f6b3c00925860f9ca21c1c29c81fd4ee66fb698d68a991c"} Jul 08 18:45:43.739952 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:43.739894 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-48.ec2.internal" podStartSLOduration=3.73987331 podStartE2EDuration="3.73987331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:45:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:45:43.739523336 +0000 UTC m=+4.586798004" watchObservedRunningTime="2026-07-08 18:45:43.73987331 +0000 UTC m=+4.587147980" Jul 08 18:45:44.683084 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:44.683052 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwtbr" Jul 08 18:45:44.683604 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:44.683197 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwtbr" podUID="45fe0fb7-f912-4b72-b298-1dae0ceae220" Jul 08 18:45:44.683698 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:44.683676 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9gjk9" Jul 08 18:45:44.683811 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:44.683786 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9gjk9" podUID="02e604f5-7b5c-4aec-a685-e380d4b20a3e" Jul 08 18:45:44.683880 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:44.683846 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmgzx" Jul 08 18:45:44.683936 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:44.683916 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mmgzx" podUID="f7e42624-526e-4aaf-904f-3541dc59aeda" Jul 08 18:45:44.744436 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:44.744322 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-48.ec2.internal" event={"ID":"3d7ba020ad0d6e8fc7260c108455cdc7","Type":"ContainerStarted","Data":"ca24e652ff00a23212f8cd53ed3906006bd2a5e493aaa44b791d77332449d86d"} Jul 08 18:45:45.253926 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:45.253879 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxbtx\" (UniqueName: \"kubernetes.io/projected/f7e42624-526e-4aaf-904f-3541dc59aeda-kube-api-access-zxbtx\") pod \"network-check-target-mmgzx\" (UID: \"f7e42624-526e-4aaf-904f-3541dc59aeda\") " pod="openshift-network-diagnostics/network-check-target-mmgzx" Jul 08 18:45:45.254108 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:45.253982 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02e604f5-7b5c-4aec-a685-e380d4b20a3e-original-pull-secret\") pod \"global-pull-secret-syncer-9gjk9\" (UID: \"02e604f5-7b5c-4aec-a685-e380d4b20a3e\") " pod="kube-system/global-pull-secret-syncer-9gjk9" Jul 08 18:45:45.254108 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:45.254019 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45fe0fb7-f912-4b72-b298-1dae0ceae220-metrics-certs\") pod \"network-metrics-daemon-mwtbr\" (UID: \"45fe0fb7-f912-4b72-b298-1dae0ceae220\") " pod="openshift-multus/network-metrics-daemon-mwtbr" Jul 08 18:45:45.254234 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:45.254158 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:45:45.254234 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:45.254219 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45fe0fb7-f912-4b72-b298-1dae0ceae220-metrics-certs podName:45fe0fb7-f912-4b72-b298-1dae0ceae220 nodeName:}" failed. No retries permitted until 2026-07-08 18:45:49.254200992 +0000 UTC m=+10.101475641 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45fe0fb7-f912-4b72-b298-1dae0ceae220-metrics-certs") pod "network-metrics-daemon-mwtbr" (UID: "45fe0fb7-f912-4b72-b298-1dae0ceae220") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:45:45.254915 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:45.254704 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 18:45:45.254915 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:45.254724 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 18:45:45.254915 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:45.254738 2570 projected.go:196] Error preparing data for projected volume kube-api-access-zxbtx for pod openshift-network-diagnostics/network-check-target-mmgzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:45:45.254915 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:45.254792 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7e42624-526e-4aaf-904f-3541dc59aeda-kube-api-access-zxbtx podName:f7e42624-526e-4aaf-904f-3541dc59aeda nodeName:}" failed. No retries permitted until 2026-07-08 18:45:49.2547764 +0000 UTC m=+10.102051053 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zxbtx" (UniqueName: "kubernetes.io/projected/f7e42624-526e-4aaf-904f-3541dc59aeda-kube-api-access-zxbtx") pod "network-check-target-mmgzx" (UID: "f7e42624-526e-4aaf-904f-3541dc59aeda") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:45:45.254915 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:45.254853 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 18:45:45.254915 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:45.254886 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02e604f5-7b5c-4aec-a685-e380d4b20a3e-original-pull-secret podName:02e604f5-7b5c-4aec-a685-e380d4b20a3e nodeName:}" failed. No retries permitted until 2026-07-08 18:45:49.254873658 +0000 UTC m=+10.102148307 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/02e604f5-7b5c-4aec-a685-e380d4b20a3e-original-pull-secret") pod "global-pull-secret-syncer-9gjk9" (UID: "02e604f5-7b5c-4aec-a685-e380d4b20a3e") : object "kube-system"/"original-pull-secret" not registered Jul 08 18:45:46.684200 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:46.683868 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9gjk9" Jul 08 18:45:46.684200 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:46.683913 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmgzx" Jul 08 18:45:46.684200 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:46.683868 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwtbr" Jul 08 18:45:46.684200 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:46.684038 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9gjk9" podUID="02e604f5-7b5c-4aec-a685-e380d4b20a3e" Jul 08 18:45:46.684200 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:46.684148 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwtbr" podUID="45fe0fb7-f912-4b72-b298-1dae0ceae220" Jul 08 18:45:46.684891 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:46.684225 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mmgzx" podUID="f7e42624-526e-4aaf-904f-3541dc59aeda" Jul 08 18:45:48.684103 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:48.683767 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9gjk9" Jul 08 18:45:48.684103 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:48.683904 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9gjk9" podUID="02e604f5-7b5c-4aec-a685-e380d4b20a3e" Jul 08 18:45:48.684103 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:48.683927 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmgzx" Jul 08 18:45:48.684103 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:48.684041 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mmgzx" podUID="f7e42624-526e-4aaf-904f-3541dc59aeda" Jul 08 18:45:48.684103 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:48.684096 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwtbr" Jul 08 18:45:48.684806 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:48.684177 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwtbr" podUID="45fe0fb7-f912-4b72-b298-1dae0ceae220" Jul 08 18:45:49.289673 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:49.289631 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxbtx\" (UniqueName: \"kubernetes.io/projected/f7e42624-526e-4aaf-904f-3541dc59aeda-kube-api-access-zxbtx\") pod \"network-check-target-mmgzx\" (UID: \"f7e42624-526e-4aaf-904f-3541dc59aeda\") " pod="openshift-network-diagnostics/network-check-target-mmgzx" Jul 08 18:45:49.289868 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:49.289712 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02e604f5-7b5c-4aec-a685-e380d4b20a3e-original-pull-secret\") pod \"global-pull-secret-syncer-9gjk9\" (UID: \"02e604f5-7b5c-4aec-a685-e380d4b20a3e\") " pod="kube-system/global-pull-secret-syncer-9gjk9" Jul 08 18:45:49.289868 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:49.289748 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45fe0fb7-f912-4b72-b298-1dae0ceae220-metrics-certs\") pod \"network-metrics-daemon-mwtbr\" (UID: \"45fe0fb7-f912-4b72-b298-1dae0ceae220\") " pod="openshift-multus/network-metrics-daemon-mwtbr" Jul 08 18:45:49.289980 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:49.289871 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:45:49.289980 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:49.289935 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45fe0fb7-f912-4b72-b298-1dae0ceae220-metrics-certs podName:45fe0fb7-f912-4b72-b298-1dae0ceae220 nodeName:}" failed. No retries permitted until 2026-07-08 18:45:57.289913636 +0000 UTC m=+18.137188298 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45fe0fb7-f912-4b72-b298-1dae0ceae220-metrics-certs") pod "network-metrics-daemon-mwtbr" (UID: "45fe0fb7-f912-4b72-b298-1dae0ceae220") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:45:49.290090 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:49.290026 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 18:45:49.290090 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:49.290040 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 18:45:49.290090 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:49.290052 2570 projected.go:196] Error preparing data for projected volume kube-api-access-zxbtx for pod openshift-network-diagnostics/network-check-target-mmgzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:45:49.290090 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:49.290087 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7e42624-526e-4aaf-904f-3541dc59aeda-kube-api-access-zxbtx podName:f7e42624-526e-4aaf-904f-3541dc59aeda nodeName:}" failed. No retries permitted until 2026-07-08 18:45:57.290076526 +0000 UTC m=+18.137351169 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zxbtx" (UniqueName: "kubernetes.io/projected/f7e42624-526e-4aaf-904f-3541dc59aeda-kube-api-access-zxbtx") pod "network-check-target-mmgzx" (UID: "f7e42624-526e-4aaf-904f-3541dc59aeda") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:45:49.290330 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:49.290163 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 18:45:49.290330 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:49.290205 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02e604f5-7b5c-4aec-a685-e380d4b20a3e-original-pull-secret podName:02e604f5-7b5c-4aec-a685-e380d4b20a3e nodeName:}" failed. No retries permitted until 2026-07-08 18:45:57.290192496 +0000 UTC m=+18.137467153 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/02e604f5-7b5c-4aec-a685-e380d4b20a3e-original-pull-secret") pod "global-pull-secret-syncer-9gjk9" (UID: "02e604f5-7b5c-4aec-a685-e380d4b20a3e") : object "kube-system"/"original-pull-secret" not registered Jul 08 18:45:50.683942 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:50.683899 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwtbr" Jul 08 18:45:50.684337 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:50.684070 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwtbr" podUID="45fe0fb7-f912-4b72-b298-1dae0ceae220" Jul 08 18:45:50.684798 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:50.684539 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9gjk9" Jul 08 18:45:50.684798 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:50.684652 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9gjk9" podUID="02e604f5-7b5c-4aec-a685-e380d4b20a3e" Jul 08 18:45:50.684798 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:50.684720 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmgzx" Jul 08 18:45:50.684798 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:50.684791 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mmgzx" podUID="f7e42624-526e-4aaf-904f-3541dc59aeda" Jul 08 18:45:52.683548 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:52.683480 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9gjk9" Jul 08 18:45:52.683548 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:52.683517 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmgzx" Jul 08 18:45:52.683548 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:52.683526 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwtbr" Jul 08 18:45:52.684136 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:52.683613 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9gjk9" podUID="02e604f5-7b5c-4aec-a685-e380d4b20a3e" Jul 08 18:45:52.684136 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:52.683688 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mmgzx" podUID="f7e42624-526e-4aaf-904f-3541dc59aeda" Jul 08 18:45:52.684136 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:52.683756 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwtbr" podUID="45fe0fb7-f912-4b72-b298-1dae0ceae220" Jul 08 18:45:54.683373 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:54.683294 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwtbr" Jul 08 18:45:54.683373 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:54.683334 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9gjk9" Jul 08 18:45:54.683373 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:54.683334 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmgzx" Jul 08 18:45:54.683926 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:54.683441 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwtbr" podUID="45fe0fb7-f912-4b72-b298-1dae0ceae220" Jul 08 18:45:54.683926 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:54.683529 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9gjk9" podUID="02e604f5-7b5c-4aec-a685-e380d4b20a3e" Jul 08 18:45:54.683926 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:54.683612 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mmgzx" podUID="f7e42624-526e-4aaf-904f-3541dc59aeda" Jul 08 18:45:56.683558 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:56.683523 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9gjk9" Jul 08 18:45:56.684059 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:56.683599 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwtbr" Jul 08 18:45:56.684059 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:56.683621 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmgzx" Jul 08 18:45:56.684059 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:56.683751 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwtbr" podUID="45fe0fb7-f912-4b72-b298-1dae0ceae220" Jul 08 18:45:56.684059 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:56.683807 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mmgzx" podUID="f7e42624-526e-4aaf-904f-3541dc59aeda" Jul 08 18:45:56.684059 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:56.683875 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9gjk9" podUID="02e604f5-7b5c-4aec-a685-e380d4b20a3e" Jul 08 18:45:57.356472 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:57.356442 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02e604f5-7b5c-4aec-a685-e380d4b20a3e-original-pull-secret\") pod \"global-pull-secret-syncer-9gjk9\" (UID: \"02e604f5-7b5c-4aec-a685-e380d4b20a3e\") " pod="kube-system/global-pull-secret-syncer-9gjk9" Jul 08 18:45:57.356472 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:57.356481 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45fe0fb7-f912-4b72-b298-1dae0ceae220-metrics-certs\") pod \"network-metrics-daemon-mwtbr\" (UID: \"45fe0fb7-f912-4b72-b298-1dae0ceae220\") " pod="openshift-multus/network-metrics-daemon-mwtbr" Jul 08 18:45:57.356691 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:57.356507 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxbtx\" (UniqueName: \"kubernetes.io/projected/f7e42624-526e-4aaf-904f-3541dc59aeda-kube-api-access-zxbtx\") pod \"network-check-target-mmgzx\" (UID: \"f7e42624-526e-4aaf-904f-3541dc59aeda\") " pod="openshift-network-diagnostics/network-check-target-mmgzx" Jul 08 18:45:57.356691 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:57.356577 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 18:45:57.356691 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:57.356592 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:45:57.356691 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:57.356602 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 18:45:57.356691 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:57.356623 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 18:45:57.356691 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:57.356632 2570 projected.go:196] Error preparing data for projected volume kube-api-access-zxbtx for pod openshift-network-diagnostics/network-check-target-mmgzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:45:57.356691 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:57.356641 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02e604f5-7b5c-4aec-a685-e380d4b20a3e-original-pull-secret podName:02e604f5-7b5c-4aec-a685-e380d4b20a3e nodeName:}" failed. No retries permitted until 2026-07-08 18:46:13.356627421 +0000 UTC m=+34.203902069 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/02e604f5-7b5c-4aec-a685-e380d4b20a3e-original-pull-secret") pod "global-pull-secret-syncer-9gjk9" (UID: "02e604f5-7b5c-4aec-a685-e380d4b20a3e") : object "kube-system"/"original-pull-secret" not registered Jul 08 18:45:57.356691 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:57.356661 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45fe0fb7-f912-4b72-b298-1dae0ceae220-metrics-certs podName:45fe0fb7-f912-4b72-b298-1dae0ceae220 nodeName:}" failed. No retries permitted until 2026-07-08 18:46:13.356654336 +0000 UTC m=+34.203928980 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45fe0fb7-f912-4b72-b298-1dae0ceae220-metrics-certs") pod "network-metrics-daemon-mwtbr" (UID: "45fe0fb7-f912-4b72-b298-1dae0ceae220") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:45:57.356691 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:57.356671 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7e42624-526e-4aaf-904f-3541dc59aeda-kube-api-access-zxbtx podName:f7e42624-526e-4aaf-904f-3541dc59aeda nodeName:}" failed. No retries permitted until 2026-07-08 18:46:13.356666078 +0000 UTC m=+34.203940722 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zxbtx" (UniqueName: "kubernetes.io/projected/f7e42624-526e-4aaf-904f-3541dc59aeda-kube-api-access-zxbtx") pod "network-check-target-mmgzx" (UID: "f7e42624-526e-4aaf-904f-3541dc59aeda") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:45:58.684056 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:58.684021 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwtbr" Jul 08 18:45:58.684526 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:58.684021 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmgzx" Jul 08 18:45:58.684526 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:58.684155 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwtbr" podUID="45fe0fb7-f912-4b72-b298-1dae0ceae220" Jul 08 18:45:58.684526 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:45:58.684021 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9gjk9" Jul 08 18:45:58.684526 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:58.684208 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mmgzx" podUID="f7e42624-526e-4aaf-904f-3541dc59aeda" Jul 08 18:45:58.684526 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:45:58.684306 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9gjk9" podUID="02e604f5-7b5c-4aec-a685-e380d4b20a3e" Jul 08 18:46:00.683896 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:00.683855 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmgzx" Jul 08 18:46:00.684391 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:00.683905 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwtbr" Jul 08 18:46:00.684391 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:00.683858 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9gjk9" Jul 08 18:46:00.684391 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:00.683995 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mmgzx" podUID="f7e42624-526e-4aaf-904f-3541dc59aeda" Jul 08 18:46:00.684391 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:00.684094 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9gjk9" podUID="02e604f5-7b5c-4aec-a685-e380d4b20a3e" Jul 08 18:46:00.684391 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:00.684204 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwtbr" podUID="45fe0fb7-f912-4b72-b298-1dae0ceae220" Jul 08 18:46:01.781708 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:01.781474 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fdwjq" event={"ID":"78c9e9b1-2fc2-47f5-9966-85b349a0f009","Type":"ContainerStarted","Data":"77d846358949175dfbb93aac40a498bf2c407ef4a71b0cc80ddac927ff4150f3"} Jul 08 18:46:01.784376 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:01.783625 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8hxcn" event={"ID":"f79dc66e-2b96-42c4-9ee9-4d036cc63d3a","Type":"ContainerStarted","Data":"a7194a4b0512fa7b337993bd7e724c08469c6fe2d0ade44d4a6f7b5f7f5fb8bc"} Jul 08 18:46:01.786367 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:01.786289 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tfrw6" event={"ID":"d8fdda41-866c-40a9-95ff-599de68e4672","Type":"ContainerStarted","Data":"cdaf668a1c603b8ecc2376c128b1a894a1784f5b966f9c97c7f6734a89a8fd14"} Jul 08 18:46:01.789502 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:01.789479 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dqz5x" event={"ID":"7a28d7e6-edce-4adf-8200-d499f2b365b0","Type":"ContainerStarted","Data":"fe1add7113ea78723779617fb9e5b894db229d0fa97d87af3c9e0056f51ca4d4"} Jul 08 18:46:01.805167 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:01.805100 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-48.ec2.internal" podStartSLOduration=21.805081274 podStartE2EDuration="21.805081274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:45:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:45:44.764163114 +0000 UTC m=+5.611437777" watchObservedRunningTime="2026-07-08 18:46:01.805081274 +0000 UTC m=+22.652355941" Jul 08 18:46:01.819351 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:01.819312 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dqz5x" podStartSLOduration=3.927129447 podStartE2EDuration="22.819297821s" totalImagesPullingTime="18.892168374s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:46:01.819207472 +0000 UTC m=+22.666482128" watchObservedRunningTime="2026-07-08 18:46:01.819297821 +0000 UTC m=+22.666572488" Jul 08 18:46:01.836525 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:01.836486 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-8hxcn" podStartSLOduration=3.987639941 podStartE2EDuration="22.836470734s" totalImagesPullingTime="18.848830793s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:46:01.836311248 +0000 UTC m=+22.683585914" watchObservedRunningTime="2026-07-08 18:46:01.836470734 +0000 UTC m=+22.683745398" Jul 08 18:46:01.851381 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:01.851334 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tfrw6" podStartSLOduration=4.004277953 podStartE2EDuration="22.851313246s" totalImagesPullingTime="18.847035293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:46:01.85087616 +0000 UTC m=+22.698150817" watchObservedRunningTime="2026-07-08 18:46:01.851313246 +0000 UTC m=+22.698587912" Jul 08 18:46:02.683730 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:02.683535 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwtbr" Jul 08 18:46:02.683889 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:02.683540 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmgzx" Jul 08 18:46:02.683889 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:02.683830 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwtbr" podUID="45fe0fb7-f912-4b72-b298-1dae0ceae220" Jul 08 18:46:02.683889 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:02.683535 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9gjk9" Jul 08 18:46:02.683889 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:02.683873 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mmgzx" podUID="f7e42624-526e-4aaf-904f-3541dc59aeda" Jul 08 18:46:02.684038 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:02.683956 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9gjk9" podUID="02e604f5-7b5c-4aec-a685-e380d4b20a3e" Jul 08 18:46:02.793686 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:02.793654 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfb5g" event={"ID":"5377c392-5ccf-4e87-ba8f-69a4fc166088","Type":"ContainerStarted","Data":"217036bc1c84797634752d69128bd297c471a397d7e52b746bcf3a3e0b12142c"} Jul 08 18:46:02.795244 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:02.795215 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" event={"ID":"1b90c2de-d63a-429c-8a95-7de30dc0b0ed","Type":"ContainerStarted","Data":"ae5e6e90043c804e11548618f24cc94ab23174fab509ec7ed714eb2a0f909fc4"} Jul 08 18:46:02.798703 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:02.798672 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" event={"ID":"c95d4430-b571-42e1-b016-686b9aa91245","Type":"ContainerStarted","Data":"3f0a2976900ee8503126726d5d9cb9daf63821c3e4d2a05fbca21d59288989ae"} Jul 08 18:46:02.798808 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:02.798754 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" event={"ID":"c95d4430-b571-42e1-b016-686b9aa91245","Type":"ContainerStarted","Data":"ce41e5f5e731dda9abcfd971ad520c15fc52b5f94d84af5424e8eb9270af5bd3"} Jul 08 18:46:02.798808 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:02.798794 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" event={"ID":"c95d4430-b571-42e1-b016-686b9aa91245","Type":"ContainerStarted","Data":"bc17d3891ef82b4706a625dc9f15dae29511caec58ee4da1f4a4fa2d4776d27b"} Jul 08 18:46:02.798928 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:02.798842 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" event={"ID":"c95d4430-b571-42e1-b016-686b9aa91245","Type":"ContainerStarted","Data":"9b8989d42a79c78de463a7bd49e5b9f0e8cabb4ec1810b5b94b94621d040a255"} Jul 08 18:46:02.798928 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:02.798875 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" event={"ID":"c95d4430-b571-42e1-b016-686b9aa91245","Type":"ContainerStarted","Data":"6ca468ad89c99efb2881e42da33fcbc48c39019e8a4b3c2d53d882c5ad8a7713"} Jul 08 18:46:02.798928 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:02.798901 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" event={"ID":"c95d4430-b571-42e1-b016-686b9aa91245","Type":"ContainerStarted","Data":"f1688b2d3afc627d2c7cae454d1623f7c11ca41bfd5328878f72f879adb8ab08"} Jul 08 18:46:02.800368 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:02.800324 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c2vbf" event={"ID":"1e1cf566-6a03-4273-a870-5f78662ca68c","Type":"ContainerStarted","Data":"afd59d0eb05f4b40f8d0c09952f32c7cd74c95347478a1a05162a4e87478c070"} Jul 08 18:46:02.801973 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:02.801948 2570 generic.go:356] "Generic (PLEG): container finished" podID="78c9e9b1-2fc2-47f5-9966-85b349a0f009" containerID="77d846358949175dfbb93aac40a498bf2c407ef4a71b0cc80ddac927ff4150f3" exitCode=0 Jul 08 18:46:02.802077 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:02.802041 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fdwjq" event={"ID":"78c9e9b1-2fc2-47f5-9966-85b349a0f009","Type":"ContainerDied","Data":"77d846358949175dfbb93aac40a498bf2c407ef4a71b0cc80ddac927ff4150f3"} Jul 08 18:46:02.823409 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:02.823354 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2jcvm" podStartSLOduration=4.927647335 podStartE2EDuration="23.823336469s" totalImagesPullingTime="18.895689134s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:46:02.823039693 +0000 UTC m=+23.670314364" watchObservedRunningTime="2026-07-08 18:46:02.823336469 +0000 UTC m=+23.670611136" Jul 08 18:46:02.878012 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:02.877959 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-c2vbf" podStartSLOduration=4.766686297 podStartE2EDuration="23.877943176s" totalImagesPullingTime="19.111256879s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:46:02.877644685 +0000 UTC m=+23.724919353" watchObservedRunningTime="2026-07-08 18:46:02.877943176 +0000 UTC m=+23.725217844" Jul 08 18:46:03.248570 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:03.248540 2570 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 08 18:46:03.449774 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:03.449707 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tfrw6" Jul 08 18:46:03.450235 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:03.450221 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tfrw6" Jul 08 18:46:03.584162 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:03.584067 2570 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-08T18:46:03.248566775Z","UUID":"d65c0f4b-59eb-408b-9513-26c0640b92aa","Handler":null,"Name":"","Endpoint":""} Jul 08 18:46:03.585518 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:03.585497 2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 08 18:46:03.585630 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:03.585533 2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 08 18:46:03.807143 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:03.807057 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5njd8" event={"ID":"d7c69034-21a4-43da-ab51-09b49ac4a67a","Type":"ContainerStarted","Data":"501e4aaeae917e30e16e200d3fc5d21b6e7ab8cf9f65de740d1370708f72d95f"} Jul 08 18:46:03.809710 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:03.809299 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfb5g" event={"ID":"5377c392-5ccf-4e87-ba8f-69a4fc166088","Type":"ContainerStarted","Data":"de09969fa03025218759e04d70cf82ab26b9f03b108810a0e69d6fadb6a07dad"} Jul 08 18:46:03.821866 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:03.821819 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5njd8" podStartSLOduration=5.9190785120000005 podStartE2EDuration="24.821778532s" totalImagesPullingTime="18.90270002s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:46:03.82162559 +0000 UTC m=+24.668900257" watchObservedRunningTime="2026-07-08 18:46:03.821778532 +0000 UTC m=+24.669053200" Jul 08 18:46:04.683535 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:04.683499 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwtbr" Jul 08 18:46:04.683535 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:04.683532 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9gjk9" Jul 08 18:46:04.683841 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:04.683510 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmgzx" Jul 08 18:46:04.683841 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:04.683638 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwtbr" podUID="45fe0fb7-f912-4b72-b298-1dae0ceae220" Jul 08 18:46:04.683841 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:04.683709 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mmgzx" podUID="f7e42624-526e-4aaf-904f-3541dc59aeda" Jul 08 18:46:04.683841 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:04.683814 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9gjk9" podUID="02e604f5-7b5c-4aec-a685-e380d4b20a3e" Jul 08 18:46:04.815402 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:04.815329 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" event={"ID":"c95d4430-b571-42e1-b016-686b9aa91245","Type":"ContainerStarted","Data":"e5dbb4b30f9215600e4cc4ed04f880ae1f2702507d5c1c03885416ffc942b6be"} Jul 08 18:46:04.817147 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:04.817121 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfb5g" event={"ID":"5377c392-5ccf-4e87-ba8f-69a4fc166088","Type":"ContainerStarted","Data":"d309450b6e675a86c4dfd753499978bbd58ac9fb88b5276e1d1d6536a76d084d"} Jul 08 18:46:04.817234 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:04.817164 2570 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 08 18:46:04.840214 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:04.840168 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfb5g" podStartSLOduration=4.684803209 podStartE2EDuration="25.840154559s" totalImagesPullingTime="21.15535135s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:45:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:46:04.839790569 +0000 UTC m=+25.687065235" watchObservedRunningTime="2026-07-08 18:46:04.840154559 +0000 UTC m=+25.687429224" Jul 08 18:46:05.497358 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:05.497125 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tfrw6" Jul 08 18:46:05.497765 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:05.497714 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tfrw6" Jul 08 18:46:06.683821 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:06.683367 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmgzx" Jul 08 18:46:06.683821 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:06.683400 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwtbr" Jul 08 18:46:06.683821 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:06.683494 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9gjk9" Jul 08 18:46:06.683821 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:06.683498 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mmgzx" podUID="f7e42624-526e-4aaf-904f-3541dc59aeda" Jul 08 18:46:06.683821 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:06.683571 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwtbr" podUID="45fe0fb7-f912-4b72-b298-1dae0ceae220" Jul 08 18:46:06.683821 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:06.683657 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9gjk9" podUID="02e604f5-7b5c-4aec-a685-e380d4b20a3e" Jul 08 18:46:06.826514 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:06.826421 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" event={"ID":"c95d4430-b571-42e1-b016-686b9aa91245","Type":"ContainerStarted","Data":"c8037dd28343cbd6c86ba53516386b2ad6980a90c5d2e749a1877d6739c34d65"} Jul 08 18:46:06.860132 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:06.857815 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" podStartSLOduration=8.752469867 podStartE2EDuration="27.857794537s" totalImagesPullingTime="19.10532467s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:46:06.856951542 +0000 UTC m=+27.704226207" watchObservedRunningTime="2026-07-08 18:46:06.857794537 +0000 UTC m=+27.705069204" Jul 08 18:46:07.830368 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:07.830124 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:46:07.830967 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:07.830408 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:46:07.830967 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:07.830447 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:46:07.853756 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:07.853725 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:46:07.854506 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:07.854480 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:46:08.599274 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:08.599216 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mmgzx"] Jul 08 18:46:08.599464 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:08.599373 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmgzx" Jul 08 18:46:08.599525 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:08.599481 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mmgzx" podUID="f7e42624-526e-4aaf-904f-3541dc59aeda" Jul 08 18:46:08.599934 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:08.599906 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9gjk9"] Jul 08 18:46:08.600091 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:08.600016 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9gjk9" Jul 08 18:46:08.600187 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:08.600143 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9gjk9" podUID="02e604f5-7b5c-4aec-a685-e380d4b20a3e" Jul 08 18:46:08.601075 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:08.601052 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mwtbr"] Jul 08 18:46:08.601175 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:08.601157 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwtbr" Jul 08 18:46:08.601249 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:08.601233 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwtbr" podUID="45fe0fb7-f912-4b72-b298-1dae0ceae220" Jul 08 18:46:09.684631 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:09.684589 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9gjk9" Jul 08 18:46:09.684996 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:09.684690 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9gjk9" podUID="02e604f5-7b5c-4aec-a685-e380d4b20a3e" Jul 08 18:46:09.684996 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:09.684693 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmgzx" Jul 08 18:46:09.684996 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:09.684749 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mmgzx" podUID="f7e42624-526e-4aaf-904f-3541dc59aeda" Jul 08 18:46:09.835155 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:09.835115 2570 generic.go:356] "Generic (PLEG): container finished" podID="78c9e9b1-2fc2-47f5-9966-85b349a0f009" containerID="a9c03c661065fdf7dd0db1b8438c9299db41a70beb3e6cfe07991a686816fdbc" exitCode=0 Jul 08 18:46:09.835331 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:09.835196 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fdwjq" event={"ID":"78c9e9b1-2fc2-47f5-9966-85b349a0f009","Type":"ContainerDied","Data":"a9c03c661065fdf7dd0db1b8438c9299db41a70beb3e6cfe07991a686816fdbc"} Jul 08 18:46:10.683903 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:10.683861 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwtbr" Jul 08 18:46:10.684068 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:10.683988 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwtbr" podUID="45fe0fb7-f912-4b72-b298-1dae0ceae220" Jul 08 18:46:11.683999 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:11.683961 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmgzx" Jul 08 18:46:11.684461 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:11.684070 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mmgzx" podUID="f7e42624-526e-4aaf-904f-3541dc59aeda" Jul 08 18:46:11.684461 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:11.684134 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9gjk9" Jul 08 18:46:11.684461 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:11.684271 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9gjk9" podUID="02e604f5-7b5c-4aec-a685-e380d4b20a3e" Jul 08 18:46:11.841748 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:11.841713 2570 generic.go:356] "Generic (PLEG): container finished" podID="78c9e9b1-2fc2-47f5-9966-85b349a0f009" containerID="9fe28a4af042e3d9630a0dc54bc22e1e3ed99e85e73f3683628f3469c2968aff" exitCode=0 Jul 08 18:46:11.841905 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:11.841792 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fdwjq" event={"ID":"78c9e9b1-2fc2-47f5-9966-85b349a0f009","Type":"ContainerDied","Data":"9fe28a4af042e3d9630a0dc54bc22e1e3ed99e85e73f3683628f3469c2968aff"} Jul 08 18:46:12.683479 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:12.683442 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwtbr" Jul 08 18:46:12.683622 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:12.683557 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwtbr" podUID="45fe0fb7-f912-4b72-b298-1dae0ceae220" Jul 08 18:46:12.846170 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:12.846086 2570 generic.go:356] "Generic (PLEG): container finished" podID="78c9e9b1-2fc2-47f5-9966-85b349a0f009" containerID="0b9f5fd5f9aff30a30024e8526ffd81fa0381f11d8e0c061e8f021121a170f69" exitCode=0 Jul 08 18:46:12.846170 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:12.846134 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fdwjq" event={"ID":"78c9e9b1-2fc2-47f5-9966-85b349a0f009","Type":"ContainerDied","Data":"0b9f5fd5f9aff30a30024e8526ffd81fa0381f11d8e0c061e8f021121a170f69"} Jul 08 18:46:13.375034 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:13.374995 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02e604f5-7b5c-4aec-a685-e380d4b20a3e-original-pull-secret\") pod \"global-pull-secret-syncer-9gjk9\" (UID: \"02e604f5-7b5c-4aec-a685-e380d4b20a3e\") " pod="kube-system/global-pull-secret-syncer-9gjk9" Jul 08 18:46:13.375240 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:13.375052 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45fe0fb7-f912-4b72-b298-1dae0ceae220-metrics-certs\") pod \"network-metrics-daemon-mwtbr\" (UID: \"45fe0fb7-f912-4b72-b298-1dae0ceae220\") " pod="openshift-multus/network-metrics-daemon-mwtbr" Jul 08 18:46:13.375240 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:13.375093 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxbtx\" (UniqueName: \"kubernetes.io/projected/f7e42624-526e-4aaf-904f-3541dc59aeda-kube-api-access-zxbtx\") pod \"network-check-target-mmgzx\" (UID: \"f7e42624-526e-4aaf-904f-3541dc59aeda\") " pod="openshift-network-diagnostics/network-check-target-mmgzx" Jul 08 18:46:13.375240 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:13.375160 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 18:46:13.375240 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:13.375182 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:46:13.375240 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:13.375227 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 18:46:13.375240 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:13.375244 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02e604f5-7b5c-4aec-a685-e380d4b20a3e-original-pull-secret podName:02e604f5-7b5c-4aec-a685-e380d4b20a3e nodeName:}" failed. No retries permitted until 2026-07-08 18:46:45.375222631 +0000 UTC m=+66.222497275 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/02e604f5-7b5c-4aec-a685-e380d4b20a3e-original-pull-secret") pod "global-pull-secret-syncer-9gjk9" (UID: "02e604f5-7b5c-4aec-a685-e380d4b20a3e") : object "kube-system"/"original-pull-secret" not registered Jul 08 18:46:13.375240 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:13.375244 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 18:46:13.375687 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:13.375279 2570 projected.go:196] Error preparing data for projected volume kube-api-access-zxbtx for pod openshift-network-diagnostics/network-check-target-mmgzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:46:13.375687 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:13.375290 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45fe0fb7-f912-4b72-b298-1dae0ceae220-metrics-certs podName:45fe0fb7-f912-4b72-b298-1dae0ceae220 nodeName:}" failed. No retries permitted until 2026-07-08 18:46:45.375273416 +0000 UTC m=+66.222548060 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45fe0fb7-f912-4b72-b298-1dae0ceae220-metrics-certs") pod "network-metrics-daemon-mwtbr" (UID: "45fe0fb7-f912-4b72-b298-1dae0ceae220") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:46:13.375687 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:13.375309 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7e42624-526e-4aaf-904f-3541dc59aeda-kube-api-access-zxbtx podName:f7e42624-526e-4aaf-904f-3541dc59aeda nodeName:}" failed. No retries permitted until 2026-07-08 18:46:45.375301887 +0000 UTC m=+66.222576532 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zxbtx" (UniqueName: "kubernetes.io/projected/f7e42624-526e-4aaf-904f-3541dc59aeda-kube-api-access-zxbtx") pod "network-check-target-mmgzx" (UID: "f7e42624-526e-4aaf-904f-3541dc59aeda") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:46:13.683910 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:13.683824 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9gjk9" Jul 08 18:46:13.683910 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:13.683837 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmgzx" Jul 08 18:46:13.684117 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:13.683951 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9gjk9" podUID="02e604f5-7b5c-4aec-a685-e380d4b20a3e" Jul 08 18:46:13.684117 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:13.684080 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mmgzx" podUID="f7e42624-526e-4aaf-904f-3541dc59aeda" Jul 08 18:46:14.425241 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.425209 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-48.ec2.internal" event="NodeReady" Jul 08 18:46:14.425698 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.425351 2570 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 08 18:46:14.482770 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.482691 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-66d5c86666-wqq74"] Jul 08 18:46:14.484893 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.484872 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66d5c86666-wqq74" Jul 08 18:46:14.488058 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.487993 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 08 18:46:14.488492 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.488470 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 08 18:46:14.488594 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.488478 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sbfpk\"" Jul 08 18:46:14.488870 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.488824 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 08 18:46:14.499215 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.499159 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 08 18:46:14.499489 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.499473 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rhwrh"] Jul 08 18:46:14.502093 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.502073 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kg6bp"] Jul 08 18:46:14.502203 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.502185 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rhwrh" Jul 08 18:46:14.503710 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.503692 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kg6bp" Jul 08 18:46:14.506557 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.506525 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 08 18:46:14.506768 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.506622 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k5jtp\"" Jul 08 18:46:14.507831 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.507669 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 08 18:46:14.507831 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.507684 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 08 18:46:14.507960 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.507867 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 08 18:46:14.508212 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.508187 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2cgrn\"" Jul 08 18:46:14.508991 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.508973 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66d5c86666-wqq74"] Jul 08 18:46:14.510267 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.510236 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 08 18:46:14.519776 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.519719 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rhwrh"] Jul 08 18:46:14.520601 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.520577 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kg6bp"] Jul 08 18:46:14.583705 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.583675 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f0490c6-dc1a-4e6e-8e53-e9a0eb220119-tmp-dir\") pod \"dns-default-rhwrh\" (UID: \"5f0490c6-dc1a-4e6e-8e53-e9a0eb220119\") " pod="openshift-dns/dns-default-rhwrh" Jul 08 18:46:14.583847 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.583716 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-registry-tls\") pod \"image-registry-66d5c86666-wqq74\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " pod="openshift-image-registry/image-registry-66d5c86666-wqq74" Jul 08 18:46:14.583847 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.583743 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-registry-certificates\") pod \"image-registry-66d5c86666-wqq74\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " pod="openshift-image-registry/image-registry-66d5c86666-wqq74" Jul 08 18:46:14.583847 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.583769 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f0490c6-dc1a-4e6e-8e53-e9a0eb220119-metrics-tls\") pod \"dns-default-rhwrh\" (UID: \"5f0490c6-dc1a-4e6e-8e53-e9a0eb220119\") " pod="openshift-dns/dns-default-rhwrh" Jul 08 18:46:14.583847 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.583797 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-image-registry-private-configuration\") pod \"image-registry-66d5c86666-wqq74\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " pod="openshift-image-registry/image-registry-66d5c86666-wqq74" Jul 08 18:46:14.583847 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.583845 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-ca-trust-extracted\") pod \"image-registry-66d5c86666-wqq74\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " pod="openshift-image-registry/image-registry-66d5c86666-wqq74" Jul 08 18:46:14.584026 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.583888 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2ccx\" (UniqueName: \"kubernetes.io/projected/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-kube-api-access-v2ccx\") pod \"image-registry-66d5c86666-wqq74\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " pod="openshift-image-registry/image-registry-66d5c86666-wqq74" Jul 08 18:46:14.584026 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.583904 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv8dz\" (UniqueName: \"kubernetes.io/projected/bcc10be2-d0a7-47a2-9b49-0316b50cb8da-kube-api-access-sv8dz\") pod \"ingress-canary-kg6bp\" (UID: \"bcc10be2-d0a7-47a2-9b49-0316b50cb8da\") " pod="openshift-ingress-canary/ingress-canary-kg6bp" Jul 08 18:46:14.584026 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.583928 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-trusted-ca\") pod \"image-registry-66d5c86666-wqq74\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " pod="openshift-image-registry/image-registry-66d5c86666-wqq74" Jul 08 18:46:14.584026 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.583951 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-bound-sa-token\") pod \"image-registry-66d5c86666-wqq74\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " pod="openshift-image-registry/image-registry-66d5c86666-wqq74" Jul 08 18:46:14.584026 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.584003 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-227np\" (UniqueName: \"kubernetes.io/projected/5f0490c6-dc1a-4e6e-8e53-e9a0eb220119-kube-api-access-227np\") pod \"dns-default-rhwrh\" (UID: \"5f0490c6-dc1a-4e6e-8e53-e9a0eb220119\") " pod="openshift-dns/dns-default-rhwrh" Jul 08 18:46:14.584200 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.584039 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5f0490c6-dc1a-4e6e-8e53-e9a0eb220119-config-volume\") pod \"dns-default-rhwrh\" (UID: \"5f0490c6-dc1a-4e6e-8e53-e9a0eb220119\") " pod="openshift-dns/dns-default-rhwrh" Jul 08 18:46:14.584200 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.584062 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-installation-pull-secrets\") pod \"image-registry-66d5c86666-wqq74\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " pod="openshift-image-registry/image-registry-66d5c86666-wqq74" Jul 08 18:46:14.584200 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.584077 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bcc10be2-d0a7-47a2-9b49-0316b50cb8da-cert\") pod \"ingress-canary-kg6bp\" (UID: \"bcc10be2-d0a7-47a2-9b49-0316b50cb8da\") " pod="openshift-ingress-canary/ingress-canary-kg6bp" Jul 08 18:46:14.683485 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.683454 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwtbr" Jul 08 18:46:14.684496 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.684472 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-ca-trust-extracted\") pod \"image-registry-66d5c86666-wqq74\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " pod="openshift-image-registry/image-registry-66d5c86666-wqq74" Jul 08 18:46:14.684594 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.684513 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2ccx\" (UniqueName: \"kubernetes.io/projected/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-kube-api-access-v2ccx\") pod \"image-registry-66d5c86666-wqq74\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " pod="openshift-image-registry/image-registry-66d5c86666-wqq74" Jul 08 18:46:14.684594 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.684532 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv8dz\" (UniqueName: \"kubernetes.io/projected/bcc10be2-d0a7-47a2-9b49-0316b50cb8da-kube-api-access-sv8dz\") pod \"ingress-canary-kg6bp\" (UID: \"bcc10be2-d0a7-47a2-9b49-0316b50cb8da\") " pod="openshift-ingress-canary/ingress-canary-kg6bp" Jul 08 18:46:14.684594 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.684560 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-trusted-ca\") pod \"image-registry-66d5c86666-wqq74\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " pod="openshift-image-registry/image-registry-66d5c86666-wqq74" Jul 08 18:46:14.684594 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.684587 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-bound-sa-token\") pod \"image-registry-66d5c86666-wqq74\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " pod="openshift-image-registry/image-registry-66d5c86666-wqq74" Jul 08 18:46:14.684857 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.684624 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-227np\" (UniqueName: \"kubernetes.io/projected/5f0490c6-dc1a-4e6e-8e53-e9a0eb220119-kube-api-access-227np\") pod \"dns-default-rhwrh\" (UID: \"5f0490c6-dc1a-4e6e-8e53-e9a0eb220119\") " pod="openshift-dns/dns-default-rhwrh" Jul 08 18:46:14.684857 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.684661 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5f0490c6-dc1a-4e6e-8e53-e9a0eb220119-config-volume\") pod \"dns-default-rhwrh\" (UID: \"5f0490c6-dc1a-4e6e-8e53-e9a0eb220119\") " pod="openshift-dns/dns-default-rhwrh" Jul 08 18:46:14.684857 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.684685 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-installation-pull-secrets\") pod \"image-registry-66d5c86666-wqq74\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " pod="openshift-image-registry/image-registry-66d5c86666-wqq74" Jul 08 18:46:14.684857 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.684708 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bcc10be2-d0a7-47a2-9b49-0316b50cb8da-cert\") pod \"ingress-canary-kg6bp\" (UID: \"bcc10be2-d0a7-47a2-9b49-0316b50cb8da\") " pod="openshift-ingress-canary/ingress-canary-kg6bp" Jul 08 18:46:14.684857 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.684745 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f0490c6-dc1a-4e6e-8e53-e9a0eb220119-tmp-dir\") pod \"dns-default-rhwrh\" (UID: \"5f0490c6-dc1a-4e6e-8e53-e9a0eb220119\") " pod="openshift-dns/dns-default-rhwrh" Jul 08 18:46:14.684857 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.684768 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-registry-tls\") pod \"image-registry-66d5c86666-wqq74\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " pod="openshift-image-registry/image-registry-66d5c86666-wqq74" Jul 08 18:46:14.684857 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.684810 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-registry-certificates\") pod \"image-registry-66d5c86666-wqq74\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " pod="openshift-image-registry/image-registry-66d5c86666-wqq74" Jul 08 18:46:14.684857 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.684838 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f0490c6-dc1a-4e6e-8e53-e9a0eb220119-metrics-tls\") pod \"dns-default-rhwrh\" (UID: \"5f0490c6-dc1a-4e6e-8e53-e9a0eb220119\") " pod="openshift-dns/dns-default-rhwrh" Jul 08 18:46:14.685217 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.684862 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-image-registry-private-configuration\") pod \"image-registry-66d5c86666-wqq74\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " pod="openshift-image-registry/image-registry-66d5c86666-wqq74" Jul 08 18:46:14.685217 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.684936 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-ca-trust-extracted\") pod \"image-registry-66d5c86666-wqq74\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " pod="openshift-image-registry/image-registry-66d5c86666-wqq74" Jul 08 18:46:14.685217 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:14.685119 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 18:46:14.685217 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:14.685179 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f0490c6-dc1a-4e6e-8e53-e9a0eb220119-metrics-tls podName:5f0490c6-dc1a-4e6e-8e53-e9a0eb220119 nodeName:}" failed. No retries permitted until 2026-07-08 18:46:15.185159053 +0000 UTC m=+36.032433700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5f0490c6-dc1a-4e6e-8e53-e9a0eb220119-metrics-tls") pod "dns-default-rhwrh" (UID: "5f0490c6-dc1a-4e6e-8e53-e9a0eb220119") : secret "dns-default-metrics-tls" not found Jul 08 18:46:14.685424 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.685319 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5f0490c6-dc1a-4e6e-8e53-e9a0eb220119-config-volume\") pod \"dns-default-rhwrh\" (UID: \"5f0490c6-dc1a-4e6e-8e53-e9a0eb220119\") " pod="openshift-dns/dns-default-rhwrh" Jul 08 18:46:14.685424 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:14.685407 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 18:46:14.685424 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:14.685421 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66d5c86666-wqq74: secret "image-registry-tls" not found Jul 08 18:46:14.685569 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:14.685463 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-registry-tls podName:2e05edf7-1d8f-4a6d-b935-4d2d91447b48 nodeName:}" failed. No retries permitted until 2026-07-08 18:46:15.185448244 +0000 UTC m=+36.032722904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-registry-tls") pod "image-registry-66d5c86666-wqq74" (UID: "2e05edf7-1d8f-4a6d-b935-4d2d91447b48") : secret "image-registry-tls" not found Jul 08 18:46:14.685653 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.685633 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f0490c6-dc1a-4e6e-8e53-e9a0eb220119-tmp-dir\") pod \"dns-default-rhwrh\" (UID: \"5f0490c6-dc1a-4e6e-8e53-e9a0eb220119\") " pod="openshift-dns/dns-default-rhwrh" Jul 08 18:46:14.685843 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.685818 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-trusted-ca\") pod \"image-registry-66d5c86666-wqq74\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " pod="openshift-image-registry/image-registry-66d5c86666-wqq74" Jul 08 18:46:14.686168 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.686145 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-registry-certificates\") pod \"image-registry-66d5c86666-wqq74\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " pod="openshift-image-registry/image-registry-66d5c86666-wqq74" Jul 08 18:46:14.686551 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.686486 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 18:46:14.686551 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.686537 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5bv4l\"" Jul 08 18:46:14.689101 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.689081 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-image-registry-private-configuration\") pod \"image-registry-66d5c86666-wqq74\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " pod="openshift-image-registry/image-registry-66d5c86666-wqq74" Jul 08 18:46:14.689177 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.689114 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-installation-pull-secrets\") pod \"image-registry-66d5c86666-wqq74\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " pod="openshift-image-registry/image-registry-66d5c86666-wqq74" Jul 08 18:46:14.689177 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.689134 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bcc10be2-d0a7-47a2-9b49-0316b50cb8da-cert\") pod \"ingress-canary-kg6bp\" (UID: \"bcc10be2-d0a7-47a2-9b49-0316b50cb8da\") " pod="openshift-ingress-canary/ingress-canary-kg6bp" Jul 08 18:46:14.695246 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.695224 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-227np\" (UniqueName: \"kubernetes.io/projected/5f0490c6-dc1a-4e6e-8e53-e9a0eb220119-kube-api-access-227np\") pod \"dns-default-rhwrh\" (UID: \"5f0490c6-dc1a-4e6e-8e53-e9a0eb220119\") " pod="openshift-dns/dns-default-rhwrh" Jul 08 18:46:14.695610 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.695566 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2ccx\" (UniqueName: \"kubernetes.io/projected/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-kube-api-access-v2ccx\") pod \"image-registry-66d5c86666-wqq74\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " pod="openshift-image-registry/image-registry-66d5c86666-wqq74" Jul 08 18:46:14.695780 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.695747 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-bound-sa-token\") pod \"image-registry-66d5c86666-wqq74\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " pod="openshift-image-registry/image-registry-66d5c86666-wqq74" Jul 08 18:46:14.696783 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.696764 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv8dz\" (UniqueName: \"kubernetes.io/projected/bcc10be2-d0a7-47a2-9b49-0316b50cb8da-kube-api-access-sv8dz\") pod \"ingress-canary-kg6bp\" (UID: \"bcc10be2-d0a7-47a2-9b49-0316b50cb8da\") " pod="openshift-ingress-canary/ingress-canary-kg6bp" Jul 08 18:46:14.820540 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.820470 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kg6bp" Jul 08 18:46:14.964399 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:14.964227 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kg6bp"] Jul 08 18:46:14.967107 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:46:14.967074 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc10be2_d0a7_47a2_9b49_0316b50cb8da.slice/crio-e33ad68c88fc5f98e5c6deb8d0b639b05c164afe55a857645fd6127a5d2a2734 WatchSource:0}: Error finding container e33ad68c88fc5f98e5c6deb8d0b639b05c164afe55a857645fd6127a5d2a2734: Status 404 returned error can't find the container with id e33ad68c88fc5f98e5c6deb8d0b639b05c164afe55a857645fd6127a5d2a2734 Jul 08 18:46:15.190308 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:15.190266 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-registry-tls\") pod \"image-registry-66d5c86666-wqq74\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " pod="openshift-image-registry/image-registry-66d5c86666-wqq74" Jul 08 18:46:15.190560 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:15.190319 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f0490c6-dc1a-4e6e-8e53-e9a0eb220119-metrics-tls\") pod \"dns-default-rhwrh\" (UID: \"5f0490c6-dc1a-4e6e-8e53-e9a0eb220119\") " pod="openshift-dns/dns-default-rhwrh" Jul 08 18:46:15.190560 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:15.190373 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 18:46:15.190560 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:15.190389 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66d5c86666-wqq74: secret "image-registry-tls" not found Jul 08 18:46:15.190560 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:15.190462 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-registry-tls podName:2e05edf7-1d8f-4a6d-b935-4d2d91447b48 nodeName:}" failed. No retries permitted until 2026-07-08 18:46:16.190432947 +0000 UTC m=+37.037707615 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-registry-tls") pod "image-registry-66d5c86666-wqq74" (UID: "2e05edf7-1d8f-4a6d-b935-4d2d91447b48") : secret "image-registry-tls" not found Jul 08 18:46:15.190560 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:15.190463 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 18:46:15.190560 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:15.190527 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f0490c6-dc1a-4e6e-8e53-e9a0eb220119-metrics-tls podName:5f0490c6-dc1a-4e6e-8e53-e9a0eb220119 nodeName:}" failed. No retries permitted until 2026-07-08 18:46:16.190507104 +0000 UTC m=+37.037781748 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5f0490c6-dc1a-4e6e-8e53-e9a0eb220119-metrics-tls") pod "dns-default-rhwrh" (UID: "5f0490c6-dc1a-4e6e-8e53-e9a0eb220119") : secret "dns-default-metrics-tls" not found Jul 08 18:46:15.683696 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:15.683666 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9gjk9" Jul 08 18:46:15.684266 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:15.683844 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmgzx" Jul 08 18:46:15.687560 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:15.687535 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 18:46:15.687687 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:15.687537 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 08 18:46:15.688423 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:15.688405 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zrdnt\"" Jul 08 18:46:15.688530 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:15.688436 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 08 18:46:15.855196 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:15.855150 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kg6bp" event={"ID":"bcc10be2-d0a7-47a2-9b49-0316b50cb8da","Type":"ContainerStarted","Data":"e33ad68c88fc5f98e5c6deb8d0b639b05c164afe55a857645fd6127a5d2a2734"} Jul 08 18:46:16.199144 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:16.199109 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-registry-tls\") pod \"image-registry-66d5c86666-wqq74\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " pod="openshift-image-registry/image-registry-66d5c86666-wqq74" Jul 08 18:46:16.199336 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:16.199152 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f0490c6-dc1a-4e6e-8e53-e9a0eb220119-metrics-tls\") pod \"dns-default-rhwrh\" (UID: \"5f0490c6-dc1a-4e6e-8e53-e9a0eb220119\") " pod="openshift-dns/dns-default-rhwrh" Jul 08 18:46:16.199336 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:16.199311 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 18:46:16.199336 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:16.199318 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 18:46:16.199515 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:16.199342 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66d5c86666-wqq74: secret "image-registry-tls" not found Jul 08 18:46:16.199515 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:16.199375 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f0490c6-dc1a-4e6e-8e53-e9a0eb220119-metrics-tls podName:5f0490c6-dc1a-4e6e-8e53-e9a0eb220119 nodeName:}" failed. No retries permitted until 2026-07-08 18:46:18.199357856 +0000 UTC m=+39.046632501 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5f0490c6-dc1a-4e6e-8e53-e9a0eb220119-metrics-tls") pod "dns-default-rhwrh" (UID: "5f0490c6-dc1a-4e6e-8e53-e9a0eb220119") : secret "dns-default-metrics-tls" not found Jul 08 18:46:16.199515 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:16.199400 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-registry-tls podName:2e05edf7-1d8f-4a6d-b935-4d2d91447b48 nodeName:}" failed. No retries permitted until 2026-07-08 18:46:18.199381779 +0000 UTC m=+39.046656441 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-registry-tls") pod "image-registry-66d5c86666-wqq74" (UID: "2e05edf7-1d8f-4a6d-b935-4d2d91447b48") : secret "image-registry-tls" not found Jul 08 18:46:18.214859 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:18.214826 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-registry-tls\") pod \"image-registry-66d5c86666-wqq74\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " pod="openshift-image-registry/image-registry-66d5c86666-wqq74" Jul 08 18:46:18.215336 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:18.214867 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f0490c6-dc1a-4e6e-8e53-e9a0eb220119-metrics-tls\") pod \"dns-default-rhwrh\" (UID: \"5f0490c6-dc1a-4e6e-8e53-e9a0eb220119\") " pod="openshift-dns/dns-default-rhwrh" Jul 08 18:46:18.215336 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:18.214984 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 18:46:18.215336 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:18.214998 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 18:46:18.215336 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:18.215023 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66d5c86666-wqq74: secret "image-registry-tls" not found Jul 08 18:46:18.215336 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:18.215033 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f0490c6-dc1a-4e6e-8e53-e9a0eb220119-metrics-tls podName:5f0490c6-dc1a-4e6e-8e53-e9a0eb220119 nodeName:}" failed. No retries permitted until 2026-07-08 18:46:22.215020522 +0000 UTC m=+43.062295166 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5f0490c6-dc1a-4e6e-8e53-e9a0eb220119-metrics-tls") pod "dns-default-rhwrh" (UID: "5f0490c6-dc1a-4e6e-8e53-e9a0eb220119") : secret "dns-default-metrics-tls" not found Jul 08 18:46:18.215336 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:18.215081 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-registry-tls podName:2e05edf7-1d8f-4a6d-b935-4d2d91447b48 nodeName:}" failed. No retries permitted until 2026-07-08 18:46:22.215062775 +0000 UTC m=+43.062337423 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-registry-tls") pod "image-registry-66d5c86666-wqq74" (UID: "2e05edf7-1d8f-4a6d-b935-4d2d91447b48") : secret "image-registry-tls" not found Jul 08 18:46:18.866218 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:18.866186 2570 generic.go:356] "Generic (PLEG): container finished" podID="78c9e9b1-2fc2-47f5-9966-85b349a0f009" containerID="9c18bf40cb53471edcf6b9c19618d19e71193c48365e84f9f2e88cb076f953a7" exitCode=0 Jul 08 18:46:18.866382 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:18.866288 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fdwjq" event={"ID":"78c9e9b1-2fc2-47f5-9966-85b349a0f009","Type":"ContainerDied","Data":"9c18bf40cb53471edcf6b9c19618d19e71193c48365e84f9f2e88cb076f953a7"} Jul 08 18:46:18.867787 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:18.867759 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kg6bp" event={"ID":"bcc10be2-d0a7-47a2-9b49-0316b50cb8da","Type":"ContainerStarted","Data":"bcd0cf650b6f1941f4d474f6b3a600de9857fe2ec2847ac0318eb132fe9fd9ab"} Jul 08 18:46:18.922699 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:18.922600 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kg6bp" podStartSLOduration=1.483105784 podStartE2EDuration="4.922582953s" totalImagesPullingTime="3.439477169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:46:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:46:18.922146172 +0000 UTC m=+39.769420839" watchObservedRunningTime="2026-07-08 18:46:18.922582953 +0000 UTC m=+39.769857620" Jul 08 18:46:19.872774 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:19.872585 2570 generic.go:356] "Generic (PLEG): container finished" podID="78c9e9b1-2fc2-47f5-9966-85b349a0f009" containerID="81d7c656b7609916f983ee60683484b0098191ec5c8f6daebd5e1f9361906efb" exitCode=0 Jul 08 18:46:19.873167 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:19.872668 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fdwjq" event={"ID":"78c9e9b1-2fc2-47f5-9966-85b349a0f009","Type":"ContainerDied","Data":"81d7c656b7609916f983ee60683484b0098191ec5c8f6daebd5e1f9361906efb"} Jul 08 18:46:20.878003 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:20.877970 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fdwjq" event={"ID":"78c9e9b1-2fc2-47f5-9966-85b349a0f009","Type":"ContainerStarted","Data":"b99b25751b3032670a72ea16cc3eb5a31c88a88bd28bd8b6d3cdaa96512d71ec"} Jul 08 18:46:20.904044 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:20.903996 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fdwjq" podStartSLOduration=9.109002906 podStartE2EDuration="41.903982844s" totalImagesPullingTime="32.794979938s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:45:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:46:20.902089664 +0000 UTC m=+41.749364330" watchObservedRunningTime="2026-07-08 18:46:20.903982844 +0000 UTC m=+41.751257509" Jul 08 18:46:22.245751 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:22.245715 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-registry-tls\") pod \"image-registry-66d5c86666-wqq74\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " pod="openshift-image-registry/image-registry-66d5c86666-wqq74" Jul 08 18:46:22.245751 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:22.245754 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f0490c6-dc1a-4e6e-8e53-e9a0eb220119-metrics-tls\") pod \"dns-default-rhwrh\" (UID: \"5f0490c6-dc1a-4e6e-8e53-e9a0eb220119\") " pod="openshift-dns/dns-default-rhwrh" Jul 08 18:46:22.246153 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:22.245858 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 18:46:22.246153 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:22.245879 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66d5c86666-wqq74: secret "image-registry-tls" not found Jul 08 18:46:22.246153 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:22.245930 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-registry-tls podName:2e05edf7-1d8f-4a6d-b935-4d2d91447b48 nodeName:}" failed. No retries permitted until 2026-07-08 18:46:30.245914389 +0000 UTC m=+51.093189033 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-registry-tls") pod "image-registry-66d5c86666-wqq74" (UID: "2e05edf7-1d8f-4a6d-b935-4d2d91447b48") : secret "image-registry-tls" not found Jul 08 18:46:22.246153 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:22.245862 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 18:46:22.246153 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:22.245987 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f0490c6-dc1a-4e6e-8e53-e9a0eb220119-metrics-tls podName:5f0490c6-dc1a-4e6e-8e53-e9a0eb220119 nodeName:}" failed. No retries permitted until 2026-07-08 18:46:30.24597567 +0000 UTC m=+51.093250317 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5f0490c6-dc1a-4e6e-8e53-e9a0eb220119-metrics-tls") pod "dns-default-rhwrh" (UID: "5f0490c6-dc1a-4e6e-8e53-e9a0eb220119") : secret "dns-default-metrics-tls" not found Jul 08 18:46:22.822409 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:22.822382 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-db4fh"] Jul 08 18:46:22.866542 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:22.866517 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-db4fh"] Jul 08 18:46:22.866666 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:22.866626 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-db4fh" Jul 08 18:46:22.869122 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:22.869098 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 08 18:46:22.869122 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:22.869112 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 08 18:46:22.870491 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:22.870475 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 08 18:46:22.870573 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:22.870515 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4nhln\"" Jul 08 18:46:22.870629 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:22.870589 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 08 18:46:23.052247 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:23.052214 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-crio-socket\") pod \"insights-runtime-extractor-db4fh\" (UID: \"c59f953f-8ffe-4d96-9bcc-68622e7e73f7\") " pod="openshift-insights/insights-runtime-extractor-db4fh" Jul 08 18:46:23.052419 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:23.052250 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8vzf\" (UniqueName: \"kubernetes.io/projected/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-kube-api-access-b8vzf\") pod \"insights-runtime-extractor-db4fh\" (UID: \"c59f953f-8ffe-4d96-9bcc-68622e7e73f7\") " pod="openshift-insights/insights-runtime-extractor-db4fh" Jul 08 18:46:23.052419 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:23.052336 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-data-volume\") pod \"insights-runtime-extractor-db4fh\" (UID: \"c59f953f-8ffe-4d96-9bcc-68622e7e73f7\") " pod="openshift-insights/insights-runtime-extractor-db4fh" Jul 08 18:46:23.052419 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:23.052364 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-db4fh\" (UID: \"c59f953f-8ffe-4d96-9bcc-68622e7e73f7\") " pod="openshift-insights/insights-runtime-extractor-db4fh" Jul 08 18:46:23.052561 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:23.052416 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-db4fh\" (UID: \"c59f953f-8ffe-4d96-9bcc-68622e7e73f7\") " pod="openshift-insights/insights-runtime-extractor-db4fh" Jul 08 18:46:23.153597 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:23.153572 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-db4fh\" (UID: \"c59f953f-8ffe-4d96-9bcc-68622e7e73f7\") " pod="openshift-insights/insights-runtime-extractor-db4fh" Jul 08 18:46:23.153713 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:23.153644 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-crio-socket\") pod \"insights-runtime-extractor-db4fh\" (UID: \"c59f953f-8ffe-4d96-9bcc-68622e7e73f7\") " pod="openshift-insights/insights-runtime-extractor-db4fh" Jul 08 18:46:23.153713 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:23.153671 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8vzf\" (UniqueName: \"kubernetes.io/projected/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-kube-api-access-b8vzf\") pod \"insights-runtime-extractor-db4fh\" (UID: \"c59f953f-8ffe-4d96-9bcc-68622e7e73f7\") " pod="openshift-insights/insights-runtime-extractor-db4fh" Jul 08 18:46:23.153713 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:23.153700 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 18:46:23.153713 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:23.153709 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-data-volume\") pod \"insights-runtime-extractor-db4fh\" (UID: \"c59f953f-8ffe-4d96-9bcc-68622e7e73f7\") " pod="openshift-insights/insights-runtime-extractor-db4fh" Jul 08 18:46:23.153843 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:23.153778 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-kube-rbac-proxy-cm podName:c59f953f-8ffe-4d96-9bcc-68622e7e73f7 nodeName:}" failed. No retries permitted until 2026-07-08 18:46:23.653755038 +0000 UTC m=+44.501029698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-db4fh" (UID: "c59f953f-8ffe-4d96-9bcc-68622e7e73f7") : configmap "kube-rbac-proxy" not found Jul 08 18:46:23.153886 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:23.153834 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-db4fh\" (UID: \"c59f953f-8ffe-4d96-9bcc-68622e7e73f7\") " pod="openshift-insights/insights-runtime-extractor-db4fh" Jul 08 18:46:23.153926 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:23.153882 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-crio-socket\") pod \"insights-runtime-extractor-db4fh\" (UID: \"c59f953f-8ffe-4d96-9bcc-68622e7e73f7\") " pod="openshift-insights/insights-runtime-extractor-db4fh" Jul 08 18:46:23.153959 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:23.153944 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 18:46:23.154009 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:23.153991 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-insights-runtime-extractor-tls podName:c59f953f-8ffe-4d96-9bcc-68622e7e73f7 nodeName:}" failed. No retries permitted until 2026-07-08 18:46:23.653978186 +0000 UTC m=+44.501252829 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-db4fh" (UID: "c59f953f-8ffe-4d96-9bcc-68622e7e73f7") : secret "insights-runtime-extractor-tls" not found Jul 08 18:46:23.154049 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:23.154015 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-data-volume\") pod \"insights-runtime-extractor-db4fh\" (UID: \"c59f953f-8ffe-4d96-9bcc-68622e7e73f7\") " pod="openshift-insights/insights-runtime-extractor-db4fh" Jul 08 18:46:23.166370 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:23.166350 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8vzf\" (UniqueName: \"kubernetes.io/projected/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-kube-api-access-b8vzf\") pod \"insights-runtime-extractor-db4fh\" (UID: \"c59f953f-8ffe-4d96-9bcc-68622e7e73f7\") " pod="openshift-insights/insights-runtime-extractor-db4fh" Jul 08 18:46:23.657601 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:23.657566 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-db4fh\" (UID: \"c59f953f-8ffe-4d96-9bcc-68622e7e73f7\") " pod="openshift-insights/insights-runtime-extractor-db4fh" Jul 08 18:46:23.658069 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:23.657633 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-db4fh\" (UID: \"c59f953f-8ffe-4d96-9bcc-68622e7e73f7\") " pod="openshift-insights/insights-runtime-extractor-db4fh" Jul 08 18:46:23.658069 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:23.657725 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 18:46:23.658069 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:23.657792 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-kube-rbac-proxy-cm podName:c59f953f-8ffe-4d96-9bcc-68622e7e73f7 nodeName:}" failed. No retries permitted until 2026-07-08 18:46:24.657772798 +0000 UTC m=+45.505047444 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-db4fh" (UID: "c59f953f-8ffe-4d96-9bcc-68622e7e73f7") : configmap "kube-rbac-proxy" not found Jul 08 18:46:23.658069 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:23.657727 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 18:46:23.658069 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:23.657858 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-insights-runtime-extractor-tls podName:c59f953f-8ffe-4d96-9bcc-68622e7e73f7 nodeName:}" failed. No retries permitted until 2026-07-08 18:46:24.657843999 +0000 UTC m=+45.505118653 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-db4fh" (UID: "c59f953f-8ffe-4d96-9bcc-68622e7e73f7") : secret "insights-runtime-extractor-tls" not found Jul 08 18:46:24.664582 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:24.664546 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-db4fh\" (UID: \"c59f953f-8ffe-4d96-9bcc-68622e7e73f7\") " pod="openshift-insights/insights-runtime-extractor-db4fh" Jul 08 18:46:24.664952 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:24.664606 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-db4fh\" (UID: \"c59f953f-8ffe-4d96-9bcc-68622e7e73f7\") " pod="openshift-insights/insights-runtime-extractor-db4fh" Jul 08 18:46:24.664952 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:24.664683 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 18:46:24.664952 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:24.664692 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 18:46:24.664952 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:24.664745 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-insights-runtime-extractor-tls podName:c59f953f-8ffe-4d96-9bcc-68622e7e73f7 nodeName:}" failed. No retries permitted until 2026-07-08 18:46:26.664731386 +0000 UTC m=+47.512006030 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-db4fh" (UID: "c59f953f-8ffe-4d96-9bcc-68622e7e73f7") : secret "insights-runtime-extractor-tls" not found Jul 08 18:46:24.664952 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:24.664757 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-kube-rbac-proxy-cm podName:c59f953f-8ffe-4d96-9bcc-68622e7e73f7 nodeName:}" failed. No retries permitted until 2026-07-08 18:46:26.664751213 +0000 UTC m=+47.512025857 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-db4fh" (UID: "c59f953f-8ffe-4d96-9bcc-68622e7e73f7") : configmap "kube-rbac-proxy" not found Jul 08 18:46:25.400441 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:25.400411 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8hxcn_f79dc66e-2b96-42c4-9ee9-4d036cc63d3a/dns-node-resolver/0.log" Jul 08 18:46:26.678268 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:26.678220 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-bbrn2"] Jul 08 18:46:26.680567 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:26.680545 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-db4fh\" (UID: \"c59f953f-8ffe-4d96-9bcc-68622e7e73f7\") " pod="openshift-insights/insights-runtime-extractor-db4fh" Jul 08 18:46:26.680645 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:26.680595 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-db4fh\" (UID: \"c59f953f-8ffe-4d96-9bcc-68622e7e73f7\") " pod="openshift-insights/insights-runtime-extractor-db4fh" Jul 08 18:46:26.680719 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:26.680700 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 18:46:26.680753 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:26.680744 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 18:46:26.680787 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:26.680760 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-insights-runtime-extractor-tls podName:c59f953f-8ffe-4d96-9bcc-68622e7e73f7 nodeName:}" failed. No retries permitted until 2026-07-08 18:46:30.680746003 +0000 UTC m=+51.528020650 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-db4fh" (UID: "c59f953f-8ffe-4d96-9bcc-68622e7e73f7") : secret "insights-runtime-extractor-tls" not found Jul 08 18:46:26.680787 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:26.680786 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-kube-rbac-proxy-cm podName:c59f953f-8ffe-4d96-9bcc-68622e7e73f7 nodeName:}" failed. No retries permitted until 2026-07-08 18:46:30.680774468 +0000 UTC m=+51.528049115 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-db4fh" (UID: "c59f953f-8ffe-4d96-9bcc-68622e7e73f7") : configmap "kube-rbac-proxy" not found Jul 08 18:46:26.701741 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:26.701718 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-bbrn2"] Jul 08 18:46:26.701854 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:26.701813 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-bbrn2" Jul 08 18:46:26.704655 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:26.704633 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 08 18:46:26.704808 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:26.704722 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 08 18:46:26.704870 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:26.704856 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-bjhqf\"" Jul 08 18:46:26.762771 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:26.762746 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-4wjvz"] Jul 08 18:46:26.784922 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:26.784901 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-4wjvz"] Jul 08 18:46:26.785026 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:26.784986 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-4wjvz" Jul 08 18:46:26.787657 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:26.787637 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 08 18:46:26.787657 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:26.787653 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 08 18:46:26.787888 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:26.787872 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 08 18:46:26.787960 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:26.787923 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-hcvkd\"" Jul 08 18:46:26.788030 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:26.787976 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 08 18:46:26.803566 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:26.803545 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dqz5x_7a28d7e6-edce-4adf-8200-d499f2b365b0/node-ca/0.log" Jul 08 18:46:26.882154 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:26.882130 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ed038c0a-ac37-4723-abee-8b820d63e4bd-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-bbrn2\" (UID: \"ed038c0a-ac37-4723-abee-8b820d63e4bd\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-bbrn2" Jul 08 18:46:26.882274 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:26.882160 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ed038c0a-ac37-4723-abee-8b820d63e4bd-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-bbrn2\" (UID: \"ed038c0a-ac37-4723-abee-8b820d63e4bd\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-bbrn2" Jul 08 18:46:26.982970 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:26.982910 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/00b37fd7-acd7-4e2b-9f70-9ae2db212992-signing-key\") pod \"service-ca-758f599c5-4wjvz\" (UID: \"00b37fd7-acd7-4e2b-9f70-9ae2db212992\") " pod="openshift-service-ca/service-ca-758f599c5-4wjvz" Jul 08 18:46:26.982970 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:26.982953 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/00b37fd7-acd7-4e2b-9f70-9ae2db212992-signing-cabundle\") pod \"service-ca-758f599c5-4wjvz\" (UID: \"00b37fd7-acd7-4e2b-9f70-9ae2db212992\") " pod="openshift-service-ca/service-ca-758f599c5-4wjvz" Jul 08 18:46:26.983136 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:26.983010 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ed038c0a-ac37-4723-abee-8b820d63e4bd-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-bbrn2\" (UID: \"ed038c0a-ac37-4723-abee-8b820d63e4bd\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-bbrn2" Jul 08 18:46:26.983136 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:26.983048 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ed038c0a-ac37-4723-abee-8b820d63e4bd-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-bbrn2\" (UID: \"ed038c0a-ac37-4723-abee-8b820d63e4bd\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-bbrn2" Jul 08 18:46:26.983136 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:26.983119 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hnhz\" (UniqueName: \"kubernetes.io/projected/00b37fd7-acd7-4e2b-9f70-9ae2db212992-kube-api-access-5hnhz\") pod \"service-ca-758f599c5-4wjvz\" (UID: \"00b37fd7-acd7-4e2b-9f70-9ae2db212992\") " pod="openshift-service-ca/service-ca-758f599c5-4wjvz" Jul 08 18:46:26.983309 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:26.983142 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 18:46:26.983309 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:26.983245 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed038c0a-ac37-4723-abee-8b820d63e4bd-networking-console-plugin-cert podName:ed038c0a-ac37-4723-abee-8b820d63e4bd nodeName:}" failed. No retries permitted until 2026-07-08 18:46:27.48322613 +0000 UTC m=+48.330500791 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ed038c0a-ac37-4723-abee-8b820d63e4bd-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-bbrn2" (UID: "ed038c0a-ac37-4723-abee-8b820d63e4bd") : secret "networking-console-plugin-cert" not found Jul 08 18:46:26.983567 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:26.983551 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ed038c0a-ac37-4723-abee-8b820d63e4bd-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-bbrn2\" (UID: \"ed038c0a-ac37-4723-abee-8b820d63e4bd\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-bbrn2" Jul 08 18:46:27.083670 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:27.083643 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hnhz\" (UniqueName: \"kubernetes.io/projected/00b37fd7-acd7-4e2b-9f70-9ae2db212992-kube-api-access-5hnhz\") pod \"service-ca-758f599c5-4wjvz\" (UID: \"00b37fd7-acd7-4e2b-9f70-9ae2db212992\") " pod="openshift-service-ca/service-ca-758f599c5-4wjvz" Jul 08 18:46:27.083802 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:27.083722 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/00b37fd7-acd7-4e2b-9f70-9ae2db212992-signing-key\") pod \"service-ca-758f599c5-4wjvz\" (UID: \"00b37fd7-acd7-4e2b-9f70-9ae2db212992\") " pod="openshift-service-ca/service-ca-758f599c5-4wjvz" Jul 08 18:46:27.083802 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:27.083769 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/00b37fd7-acd7-4e2b-9f70-9ae2db212992-signing-cabundle\") pod \"service-ca-758f599c5-4wjvz\" (UID: \"00b37fd7-acd7-4e2b-9f70-9ae2db212992\") " pod="openshift-service-ca/service-ca-758f599c5-4wjvz" Jul 08 18:46:27.084366 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:27.084338 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/00b37fd7-acd7-4e2b-9f70-9ae2db212992-signing-cabundle\") pod \"service-ca-758f599c5-4wjvz\" (UID: \"00b37fd7-acd7-4e2b-9f70-9ae2db212992\") " pod="openshift-service-ca/service-ca-758f599c5-4wjvz" Jul 08 18:46:27.086091 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:27.086068 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/00b37fd7-acd7-4e2b-9f70-9ae2db212992-signing-key\") pod \"service-ca-758f599c5-4wjvz\" (UID: \"00b37fd7-acd7-4e2b-9f70-9ae2db212992\") " pod="openshift-service-ca/service-ca-758f599c5-4wjvz" Jul 08 18:46:27.094857 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:27.094835 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hnhz\" (UniqueName: \"kubernetes.io/projected/00b37fd7-acd7-4e2b-9f70-9ae2db212992-kube-api-access-5hnhz\") pod \"service-ca-758f599c5-4wjvz\" (UID: \"00b37fd7-acd7-4e2b-9f70-9ae2db212992\") " pod="openshift-service-ca/service-ca-758f599c5-4wjvz" Jul 08 18:46:27.199772 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:27.199750 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kg6bp_bcc10be2-d0a7-47a2-9b49-0316b50cb8da/serve-healthcheck-canary/0.log" Jul 08 18:46:27.393238 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:27.393211 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-4wjvz" Jul 08 18:46:27.487539 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:27.486916 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ed038c0a-ac37-4723-abee-8b820d63e4bd-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-bbrn2\" (UID: \"ed038c0a-ac37-4723-abee-8b820d63e4bd\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-bbrn2" Jul 08 18:46:27.487539 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:27.487077 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 18:46:27.487539 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:27.487141 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed038c0a-ac37-4723-abee-8b820d63e4bd-networking-console-plugin-cert podName:ed038c0a-ac37-4723-abee-8b820d63e4bd nodeName:}" failed. No retries permitted until 2026-07-08 18:46:28.48712192 +0000 UTC m=+49.334396579 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ed038c0a-ac37-4723-abee-8b820d63e4bd-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-bbrn2" (UID: "ed038c0a-ac37-4723-abee-8b820d63e4bd") : secret "networking-console-plugin-cert" not found Jul 08 18:46:27.544527 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:27.544496 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-4wjvz"] Jul 08 18:46:27.547651 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:46:27.547619 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b37fd7_acd7_4e2b_9f70_9ae2db212992.slice/crio-18b7fd4d6af6ed62b67246c2f6a34a02a69ac8024fd0bad95db552648b4b3fcf WatchSource:0}: Error finding container 18b7fd4d6af6ed62b67246c2f6a34a02a69ac8024fd0bad95db552648b4b3fcf: Status 404 returned error can't find the container with id 18b7fd4d6af6ed62b67246c2f6a34a02a69ac8024fd0bad95db552648b4b3fcf Jul 08 18:46:27.894107 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:27.894076 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-4wjvz" event={"ID":"00b37fd7-acd7-4e2b-9f70-9ae2db212992","Type":"ContainerStarted","Data":"18b7fd4d6af6ed62b67246c2f6a34a02a69ac8024fd0bad95db552648b4b3fcf"} Jul 08 18:46:28.496404 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:28.496367 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ed038c0a-ac37-4723-abee-8b820d63e4bd-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-bbrn2\" (UID: \"ed038c0a-ac37-4723-abee-8b820d63e4bd\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-bbrn2" Jul 08 18:46:28.496579 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:28.496523 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 18:46:28.496659 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:28.496607 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed038c0a-ac37-4723-abee-8b820d63e4bd-networking-console-plugin-cert podName:ed038c0a-ac37-4723-abee-8b820d63e4bd nodeName:}" failed. No retries permitted until 2026-07-08 18:46:30.496590486 +0000 UTC m=+51.343865131 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ed038c0a-ac37-4723-abee-8b820d63e4bd-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-bbrn2" (UID: "ed038c0a-ac37-4723-abee-8b820d63e4bd") : secret "networking-console-plugin-cert" not found Jul 08 18:46:30.310696 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:30.310654 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-registry-tls\") pod \"image-registry-66d5c86666-wqq74\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " pod="openshift-image-registry/image-registry-66d5c86666-wqq74" Jul 08 18:46:30.311109 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:30.310707 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f0490c6-dc1a-4e6e-8e53-e9a0eb220119-metrics-tls\") pod \"dns-default-rhwrh\" (UID: \"5f0490c6-dc1a-4e6e-8e53-e9a0eb220119\") " pod="openshift-dns/dns-default-rhwrh" Jul 08 18:46:30.311109 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:30.310800 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 18:46:30.311109 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:30.310820 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66d5c86666-wqq74: secret "image-registry-tls" not found Jul 08 18:46:30.311109 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:30.310876 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-registry-tls podName:2e05edf7-1d8f-4a6d-b935-4d2d91447b48 nodeName:}" failed. No retries permitted until 2026-07-08 18:46:46.310860458 +0000 UTC m=+67.158135105 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-registry-tls") pod "image-registry-66d5c86666-wqq74" (UID: "2e05edf7-1d8f-4a6d-b935-4d2d91447b48") : secret "image-registry-tls" not found Jul 08 18:46:30.311109 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:30.310879 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 18:46:30.311109 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:30.310939 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f0490c6-dc1a-4e6e-8e53-e9a0eb220119-metrics-tls podName:5f0490c6-dc1a-4e6e-8e53-e9a0eb220119 nodeName:}" failed. No retries permitted until 2026-07-08 18:46:46.310926275 +0000 UTC m=+67.158200923 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5f0490c6-dc1a-4e6e-8e53-e9a0eb220119-metrics-tls") pod "dns-default-rhwrh" (UID: "5f0490c6-dc1a-4e6e-8e53-e9a0eb220119") : secret "dns-default-metrics-tls" not found Jul 08 18:46:30.511846 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:30.511819 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ed038c0a-ac37-4723-abee-8b820d63e4bd-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-bbrn2\" (UID: \"ed038c0a-ac37-4723-abee-8b820d63e4bd\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-bbrn2" Jul 08 18:46:30.511946 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:30.511922 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 18:46:30.511994 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:30.511970 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed038c0a-ac37-4723-abee-8b820d63e4bd-networking-console-plugin-cert podName:ed038c0a-ac37-4723-abee-8b820d63e4bd nodeName:}" failed. No retries permitted until 2026-07-08 18:46:34.51195813 +0000 UTC m=+55.359232775 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ed038c0a-ac37-4723-abee-8b820d63e4bd-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-bbrn2" (UID: "ed038c0a-ac37-4723-abee-8b820d63e4bd") : secret "networking-console-plugin-cert" not found Jul 08 18:46:30.713563 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:30.713530 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-db4fh\" (UID: \"c59f953f-8ffe-4d96-9bcc-68622e7e73f7\") " pod="openshift-insights/insights-runtime-extractor-db4fh" Jul 08 18:46:30.713753 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:30.713619 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-db4fh\" (UID: \"c59f953f-8ffe-4d96-9bcc-68622e7e73f7\") " pod="openshift-insights/insights-runtime-extractor-db4fh" Jul 08 18:46:30.713753 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:30.713649 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 18:46:30.713753 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:30.713700 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 18:46:30.713753 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:30.713716 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-kube-rbac-proxy-cm podName:c59f953f-8ffe-4d96-9bcc-68622e7e73f7 nodeName:}" failed. No retries permitted until 2026-07-08 18:46:38.713699765 +0000 UTC m=+59.560974411 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-db4fh" (UID: "c59f953f-8ffe-4d96-9bcc-68622e7e73f7") : configmap "kube-rbac-proxy" not found Jul 08 18:46:30.713753 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:30.713751 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-insights-runtime-extractor-tls podName:c59f953f-8ffe-4d96-9bcc-68622e7e73f7 nodeName:}" failed. No retries permitted until 2026-07-08 18:46:38.713738724 +0000 UTC m=+59.561013371 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-db4fh" (UID: "c59f953f-8ffe-4d96-9bcc-68622e7e73f7") : secret "insights-runtime-extractor-tls" not found Jul 08 18:46:30.903910 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:30.903874 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-4wjvz" event={"ID":"00b37fd7-acd7-4e2b-9f70-9ae2db212992","Type":"ContainerStarted","Data":"0b36972ba1b4ee9fdc0e9a9e1879bb7c60f2144b9694ba497ab12b61aa5d4df8"} Jul 08 18:46:30.926285 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:30.926224 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-4wjvz" podStartSLOduration=2.071786148 podStartE2EDuration="4.926210467s" totalImagesPullingTime="2.854424319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:46:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:46:30.925668008 +0000 UTC m=+51.772942676" watchObservedRunningTime="2026-07-08 18:46:30.926210467 +0000 UTC m=+51.773485156" Jul 08 18:46:34.546288 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:34.546241 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ed038c0a-ac37-4723-abee-8b820d63e4bd-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-bbrn2\" (UID: \"ed038c0a-ac37-4723-abee-8b820d63e4bd\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-bbrn2" Jul 08 18:46:34.546793 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:34.546403 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 18:46:34.546793 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:34.546480 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed038c0a-ac37-4723-abee-8b820d63e4bd-networking-console-plugin-cert podName:ed038c0a-ac37-4723-abee-8b820d63e4bd nodeName:}" failed. No retries permitted until 2026-07-08 18:46:42.546460569 +0000 UTC m=+63.393735221 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ed038c0a-ac37-4723-abee-8b820d63e4bd-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-bbrn2" (UID: "ed038c0a-ac37-4723-abee-8b820d63e4bd") : secret "networking-console-plugin-cert" not found Jul 08 18:46:38.778693 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:38.778659 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-db4fh\" (UID: \"c59f953f-8ffe-4d96-9bcc-68622e7e73f7\") " pod="openshift-insights/insights-runtime-extractor-db4fh" Jul 08 18:46:38.779153 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:38.778715 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-db4fh\" (UID: \"c59f953f-8ffe-4d96-9bcc-68622e7e73f7\") " pod="openshift-insights/insights-runtime-extractor-db4fh" Jul 08 18:46:38.779153 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:38.778820 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 18:46:38.779153 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:38.778896 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-insights-runtime-extractor-tls podName:c59f953f-8ffe-4d96-9bcc-68622e7e73f7 nodeName:}" failed. No retries permitted until 2026-07-08 18:46:54.778880222 +0000 UTC m=+75.626154869 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-db4fh" (UID: "c59f953f-8ffe-4d96-9bcc-68622e7e73f7") : secret "insights-runtime-extractor-tls" not found Jul 08 18:46:38.779292 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:38.779203 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-db4fh\" (UID: \"c59f953f-8ffe-4d96-9bcc-68622e7e73f7\") " pod="openshift-insights/insights-runtime-extractor-db4fh" Jul 08 18:46:39.845436 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:39.845408 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wqmlf" Jul 08 18:46:42.608532 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:42.608499 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ed038c0a-ac37-4723-abee-8b820d63e4bd-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-bbrn2\" (UID: \"ed038c0a-ac37-4723-abee-8b820d63e4bd\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-bbrn2" Jul 08 18:46:42.610950 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:42.610914 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ed038c0a-ac37-4723-abee-8b820d63e4bd-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-bbrn2\" (UID: \"ed038c0a-ac37-4723-abee-8b820d63e4bd\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-bbrn2" Jul 08 18:46:42.913582 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:42.913555 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-bjhqf\"" Jul 08 18:46:42.921704 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:42.921684 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-bbrn2" Jul 08 18:46:43.036924 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:43.036895 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-bbrn2"] Jul 08 18:46:43.040044 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:46:43.040014 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded038c0a_ac37_4723_abee_8b820d63e4bd.slice/crio-66dab9cd58914039b4e4b423dd589e15cc1c5bd6e9649e533501b2d437c8b2f5 WatchSource:0}: Error finding container 66dab9cd58914039b4e4b423dd589e15cc1c5bd6e9649e533501b2d437c8b2f5: Status 404 returned error can't find the container with id 66dab9cd58914039b4e4b423dd589e15cc1c5bd6e9649e533501b2d437c8b2f5 Jul 08 18:46:43.937537 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:43.937495 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-bbrn2" event={"ID":"ed038c0a-ac37-4723-abee-8b820d63e4bd","Type":"ContainerStarted","Data":"66dab9cd58914039b4e4b423dd589e15cc1c5bd6e9649e533501b2d437c8b2f5"} Jul 08 18:46:44.941600 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:44.941561 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-bbrn2" event={"ID":"ed038c0a-ac37-4723-abee-8b820d63e4bd","Type":"ContainerStarted","Data":"e1d70d4cbaf71bb397b965ab19341e7240ee4a362a296172409a2cb5b9438986"} Jul 08 18:46:44.979476 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:44.979425 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-bbrn2" podStartSLOduration=17.728181072 podStartE2EDuration="18.979407718s" totalImagesPullingTime="1.251226646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:46:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:46:44.97885556 +0000 UTC m=+65.826130236" watchObservedRunningTime="2026-07-08 18:46:44.979407718 +0000 UTC m=+65.826682384" Jul 08 18:46:45.429266 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:45.429223 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45fe0fb7-f912-4b72-b298-1dae0ceae220-metrics-certs\") pod \"network-metrics-daemon-mwtbr\" (UID: \"45fe0fb7-f912-4b72-b298-1dae0ceae220\") " pod="openshift-multus/network-metrics-daemon-mwtbr" Jul 08 18:46:45.429437 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:45.429294 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxbtx\" (UniqueName: \"kubernetes.io/projected/f7e42624-526e-4aaf-904f-3541dc59aeda-kube-api-access-zxbtx\") pod \"network-check-target-mmgzx\" (UID: \"f7e42624-526e-4aaf-904f-3541dc59aeda\") " pod="openshift-network-diagnostics/network-check-target-mmgzx" Jul 08 18:46:45.429437 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:45.429333 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02e604f5-7b5c-4aec-a685-e380d4b20a3e-original-pull-secret\") pod \"global-pull-secret-syncer-9gjk9\" (UID: \"02e604f5-7b5c-4aec-a685-e380d4b20a3e\") " pod="kube-system/global-pull-secret-syncer-9gjk9" Jul 08 18:46:45.431831 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:45.431809 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 18:46:45.432747 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:45.432730 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 18:46:45.432835 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:45.432732 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 08 18:46:45.441874 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:45.441835 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02e604f5-7b5c-4aec-a685-e380d4b20a3e-original-pull-secret\") pod \"global-pull-secret-syncer-9gjk9\" (UID: \"02e604f5-7b5c-4aec-a685-e380d4b20a3e\") " pod="kube-system/global-pull-secret-syncer-9gjk9" Jul 08 18:46:45.441951 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:45.441875 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45fe0fb7-f912-4b72-b298-1dae0ceae220-metrics-certs\") pod \"network-metrics-daemon-mwtbr\" (UID: \"45fe0fb7-f912-4b72-b298-1dae0ceae220\") " pod="openshift-multus/network-metrics-daemon-mwtbr" Jul 08 18:46:45.441989 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:45.441970 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 08 18:46:45.451821 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:45.451802 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxbtx\" (UniqueName: \"kubernetes.io/projected/f7e42624-526e-4aaf-904f-3541dc59aeda-kube-api-access-zxbtx\") pod \"network-check-target-mmgzx\" (UID: \"f7e42624-526e-4aaf-904f-3541dc59aeda\") " pod="openshift-network-diagnostics/network-check-target-mmgzx" Jul 08 18:46:45.611127 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:45.611098 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5bv4l\"" Jul 08 18:46:45.619235 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:45.619213 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwtbr" Jul 08 18:46:45.700601 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:45.700578 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9gjk9" Jul 08 18:46:45.708556 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:45.708534 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zrdnt\"" Jul 08 18:46:45.716970 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:45.716950 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmgzx" Jul 08 18:46:45.742104 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:45.742074 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mwtbr"] Jul 08 18:46:45.745674 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:46:45.745648 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45fe0fb7_f912_4b72_b298_1dae0ceae220.slice/crio-bd2e0a05f85369188f825b5c642aa563dc47d5288be2d34401ac65dc1293c980 WatchSource:0}: Error finding container bd2e0a05f85369188f825b5c642aa563dc47d5288be2d34401ac65dc1293c980: Status 404 returned error can't find the container with id bd2e0a05f85369188f825b5c642aa563dc47d5288be2d34401ac65dc1293c980 Jul 08 18:46:45.827310 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:45.827285 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9gjk9"] Jul 08 18:46:45.830587 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:46:45.830557 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02e604f5_7b5c_4aec_a685_e380d4b20a3e.slice/crio-36d494c6d23ffafb0938babb94c760514419699d80076fa800428cb420ebb376 WatchSource:0}: Error finding container 36d494c6d23ffafb0938babb94c760514419699d80076fa800428cb420ebb376: Status 404 returned error can't find the container with id 36d494c6d23ffafb0938babb94c760514419699d80076fa800428cb420ebb376 Jul 08 18:46:45.843103 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:45.843081 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mmgzx"] Jul 08 18:46:45.845841 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:46:45.845821 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e42624_526e_4aaf_904f_3541dc59aeda.slice/crio-7e4b76ce14e56f436e688beca44a270192b9bea04cad785705e530d416f705f8 WatchSource:0}: Error finding container 7e4b76ce14e56f436e688beca44a270192b9bea04cad785705e530d416f705f8: Status 404 returned error can't find the container with id 7e4b76ce14e56f436e688beca44a270192b9bea04cad785705e530d416f705f8 Jul 08 18:46:45.944606 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:45.944574 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mmgzx" event={"ID":"f7e42624-526e-4aaf-904f-3541dc59aeda","Type":"ContainerStarted","Data":"7e4b76ce14e56f436e688beca44a270192b9bea04cad785705e530d416f705f8"} Jul 08 18:46:45.945597 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:45.945575 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9gjk9" event={"ID":"02e604f5-7b5c-4aec-a685-e380d4b20a3e","Type":"ContainerStarted","Data":"36d494c6d23ffafb0938babb94c760514419699d80076fa800428cb420ebb376"} Jul 08 18:46:45.946648 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:45.946592 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mwtbr" event={"ID":"45fe0fb7-f912-4b72-b298-1dae0ceae220","Type":"ContainerStarted","Data":"bd2e0a05f85369188f825b5c642aa563dc47d5288be2d34401ac65dc1293c980"} Jul 08 18:46:46.335100 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:46.335009 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-registry-tls\") pod \"image-registry-66d5c86666-wqq74\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " pod="openshift-image-registry/image-registry-66d5c86666-wqq74" Jul 08 18:46:46.335100 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:46.335063 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f0490c6-dc1a-4e6e-8e53-e9a0eb220119-metrics-tls\") pod \"dns-default-rhwrh\" (UID: \"5f0490c6-dc1a-4e6e-8e53-e9a0eb220119\") " pod="openshift-dns/dns-default-rhwrh" Jul 08 18:46:46.337953 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:46.337923 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f0490c6-dc1a-4e6e-8e53-e9a0eb220119-metrics-tls\") pod \"dns-default-rhwrh\" (UID: \"5f0490c6-dc1a-4e6e-8e53-e9a0eb220119\") " pod="openshift-dns/dns-default-rhwrh" Jul 08 18:46:46.338072 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:46.337995 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-registry-tls\") pod \"image-registry-66d5c86666-wqq74\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " pod="openshift-image-registry/image-registry-66d5c86666-wqq74" Jul 08 18:46:46.600121 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:46.600043 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sbfpk\"" Jul 08 18:46:46.608280 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:46.608072 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66d5c86666-wqq74" Jul 08 18:46:46.617013 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:46.616779 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k5jtp\"" Jul 08 18:46:46.624752 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:46.624729 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rhwrh" Jul 08 18:46:46.893699 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:46.893304 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rhwrh"] Jul 08 18:46:46.897421 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:46:46.897390 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f0490c6_dc1a_4e6e_8e53_e9a0eb220119.slice/crio-7db13f2d4711d204f12320ed56bc340ac41b8cd1b6c7a3700d74e803873b4ec5 WatchSource:0}: Error finding container 7db13f2d4711d204f12320ed56bc340ac41b8cd1b6c7a3700d74e803873b4ec5: Status 404 returned error can't find the container with id 7db13f2d4711d204f12320ed56bc340ac41b8cd1b6c7a3700d74e803873b4ec5 Jul 08 18:46:46.913570 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:46.913507 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66d5c86666-wqq74"] Jul 08 18:46:46.917960 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:46:46.917933 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e05edf7_1d8f_4a6d_b935_4d2d91447b48.slice/crio-e6a1c93fac81df9a589aed6f23e44335968416b50a9b028301e2ec996098e8a0 WatchSource:0}: Error finding container e6a1c93fac81df9a589aed6f23e44335968416b50a9b028301e2ec996098e8a0: Status 404 returned error can't find the container with id e6a1c93fac81df9a589aed6f23e44335968416b50a9b028301e2ec996098e8a0 Jul 08 18:46:46.952815 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:46.952719 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rhwrh" event={"ID":"5f0490c6-dc1a-4e6e-8e53-e9a0eb220119","Type":"ContainerStarted","Data":"7db13f2d4711d204f12320ed56bc340ac41b8cd1b6c7a3700d74e803873b4ec5"} Jul 08 18:46:46.954982 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:46.954919 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66d5c86666-wqq74" event={"ID":"2e05edf7-1d8f-4a6d-b935-4d2d91447b48","Type":"ContainerStarted","Data":"e6a1c93fac81df9a589aed6f23e44335968416b50a9b028301e2ec996098e8a0"} Jul 08 18:46:46.957556 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:46.957487 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mwtbr" event={"ID":"45fe0fb7-f912-4b72-b298-1dae0ceae220","Type":"ContainerStarted","Data":"b534864e53c49f87462560e625f2a31b6b51c13b46cc7726772f321587b60952"} Jul 08 18:46:47.965265 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:47.965211 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mwtbr" event={"ID":"45fe0fb7-f912-4b72-b298-1dae0ceae220","Type":"ContainerStarted","Data":"dd1134037562567208f64aa8a613297cfb125fc971f4dfa91a0d1e30761c323b"} Jul 08 18:46:47.968205 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:47.968166 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66d5c86666-wqq74" event={"ID":"2e05edf7-1d8f-4a6d-b935-4d2d91447b48","Type":"ContainerStarted","Data":"fdf9546880615a231f4964457e2e5014b4fa4ccf00c2b937bed0d0cbf6bad46b"} Jul 08 18:46:47.968334 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:47.968320 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-66d5c86666-wqq74" Jul 08 18:46:48.006221 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:48.005401 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mwtbr" podStartSLOduration=67.999283039 podStartE2EDuration="1m9.005381453s" totalImagesPullingTime="1.006098414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:46:47.983577889 +0000 UTC m=+68.830852557" watchObservedRunningTime="2026-07-08 18:46:48.005381453 +0000 UTC m=+68.852656120" Jul 08 18:46:48.006221 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:48.005949 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-66d5c86666-wqq74" podStartSLOduration=54.005935342 podStartE2EDuration="54.005935342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:45:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:46:48.004463461 +0000 UTC m=+68.851738128" watchObservedRunningTime="2026-07-08 18:46:48.005935342 +0000 UTC m=+68.853210008" Jul 08 18:46:50.139828 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.139799 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7544b6849d-pd9zz"] Jul 08 18:46:50.152137 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.152116 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7544b6849d-pd9zz" Jul 08 18:46:50.155768 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.155662 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 08 18:46:50.155768 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.155710 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 08 18:46:50.155768 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.155665 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 08 18:46:50.159166 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.159148 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 08 18:46:50.174003 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.173982 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7544b6849d-pd9zz"] Jul 08 18:46:50.174781 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.174759 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66d5c86666-wqq74"] Jul 08 18:46:50.242550 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.242520 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5db8cf5545-hzsdv"] Jul 08 18:46:50.244616 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.244586 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5db8cf5545-hzsdv" Jul 08 18:46:50.246906 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.246888 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-984989766-ktxtr"] Jul 08 18:46:50.247124 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.247096 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wqtlv\"" Jul 08 18:46:50.247548 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.247523 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 08 18:46:50.247922 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.247762 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 08 18:46:50.247922 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.247808 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 08 18:46:50.247922 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.247853 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 08 18:46:50.247922 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.247882 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 08 18:46:50.248165 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.248107 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 08 18:46:50.248165 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.248155 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 08 18:46:50.248924 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.248906 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-984989766-ktxtr" Jul 08 18:46:50.252934 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.252914 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-fx4jm"] Jul 08 18:46:50.254827 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.254811 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-fx4jm" Jul 08 18:46:50.257573 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.257545 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-g72dk\"" Jul 08 18:46:50.266704 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.266682 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5ad225a-4ed3-43ba-8cdc-9d4aa1179cb2-tmp\") pod \"klusterlet-addon-workmgr-7544b6849d-pd9zz\" (UID: \"a5ad225a-4ed3-43ba-8cdc-9d4aa1179cb2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7544b6849d-pd9zz" Jul 08 18:46:50.266803 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.266733 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tcrv\" (UniqueName: \"kubernetes.io/projected/a5ad225a-4ed3-43ba-8cdc-9d4aa1179cb2-kube-api-access-6tcrv\") pod \"klusterlet-addon-workmgr-7544b6849d-pd9zz\" (UID: \"a5ad225a-4ed3-43ba-8cdc-9d4aa1179cb2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7544b6849d-pd9zz" Jul 08 18:46:50.266874 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.266842 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a5ad225a-4ed3-43ba-8cdc-9d4aa1179cb2-klusterlet-config\") pod \"klusterlet-addon-workmgr-7544b6849d-pd9zz\" (UID: \"a5ad225a-4ed3-43ba-8cdc-9d4aa1179cb2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7544b6849d-pd9zz" Jul 08 18:46:50.267766 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.267744 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-984989766-ktxtr"] Jul 08 18:46:50.277384 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.277364 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5db8cf5545-hzsdv"] Jul 08 18:46:50.288195 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.288165 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-fx4jm"] Jul 08 18:46:50.367529 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.367496 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b8f10ac-84f9-4e89-91ac-08dbc619b666-console-config\") pod \"console-5db8cf5545-hzsdv\" (UID: \"1b8f10ac-84f9-4e89-91ac-08dbc619b666\") " pod="openshift-console/console-5db8cf5545-hzsdv" Jul 08 18:46:50.367731 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.367548 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwk78\" (UniqueName: \"kubernetes.io/projected/ce0fcade-7b07-4a9a-b843-9937f2de7038-kube-api-access-kwk78\") pod \"image-registry-984989766-ktxtr\" (UID: \"ce0fcade-7b07-4a9a-b843-9937f2de7038\") " pod="openshift-image-registry/image-registry-984989766-ktxtr" Jul 08 18:46:50.367731 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.367576 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b8f10ac-84f9-4e89-91ac-08dbc619b666-oauth-serving-cert\") pod \"console-5db8cf5545-hzsdv\" (UID: \"1b8f10ac-84f9-4e89-91ac-08dbc619b666\") " pod="openshift-console/console-5db8cf5545-hzsdv" Jul 08 18:46:50.367731 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.367604 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce0fcade-7b07-4a9a-b843-9937f2de7038-trusted-ca\") pod \"image-registry-984989766-ktxtr\" (UID: \"ce0fcade-7b07-4a9a-b843-9937f2de7038\") " pod="openshift-image-registry/image-registry-984989766-ktxtr" Jul 08 18:46:50.367731 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.367682 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5ad225a-4ed3-43ba-8cdc-9d4aa1179cb2-tmp\") pod \"klusterlet-addon-workmgr-7544b6849d-pd9zz\" (UID: \"a5ad225a-4ed3-43ba-8cdc-9d4aa1179cb2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7544b6849d-pd9zz" Jul 08 18:46:50.367731 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.367721 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfps9\" (UniqueName: \"kubernetes.io/projected/d11e5bed-ffc0-483d-ad13-435bc9131d71-kube-api-access-bfps9\") pod \"downloads-685bfb5f64-fx4jm\" (UID: \"d11e5bed-ffc0-483d-ad13-435bc9131d71\") " pod="openshift-console/downloads-685bfb5f64-fx4jm" Jul 08 18:46:50.368016 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.367868 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b8f10ac-84f9-4e89-91ac-08dbc619b666-service-ca\") pod \"console-5db8cf5545-hzsdv\" (UID: \"1b8f10ac-84f9-4e89-91ac-08dbc619b666\") " pod="openshift-console/console-5db8cf5545-hzsdv" Jul 08 18:46:50.368016 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.367899 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ce0fcade-7b07-4a9a-b843-9937f2de7038-ca-trust-extracted\") pod \"image-registry-984989766-ktxtr\" (UID: \"ce0fcade-7b07-4a9a-b843-9937f2de7038\") " pod="openshift-image-registry/image-registry-984989766-ktxtr" Jul 08 18:46:50.368016 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.367931 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b8f10ac-84f9-4e89-91ac-08dbc619b666-console-serving-cert\") pod \"console-5db8cf5545-hzsdv\" (UID: \"1b8f10ac-84f9-4e89-91ac-08dbc619b666\") " pod="openshift-console/console-5db8cf5545-hzsdv" Jul 08 18:46:50.368016 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.367970 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tcrv\" (UniqueName: \"kubernetes.io/projected/a5ad225a-4ed3-43ba-8cdc-9d4aa1179cb2-kube-api-access-6tcrv\") pod \"klusterlet-addon-workmgr-7544b6849d-pd9zz\" (UID: \"a5ad225a-4ed3-43ba-8cdc-9d4aa1179cb2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7544b6849d-pd9zz" Jul 08 18:46:50.368016 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.367994 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b8f10ac-84f9-4e89-91ac-08dbc619b666-console-oauth-config\") pod \"console-5db8cf5545-hzsdv\" (UID: \"1b8f10ac-84f9-4e89-91ac-08dbc619b666\") " pod="openshift-console/console-5db8cf5545-hzsdv" Jul 08 18:46:50.368317 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.368058 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5ad225a-4ed3-43ba-8cdc-9d4aa1179cb2-tmp\") pod \"klusterlet-addon-workmgr-7544b6849d-pd9zz\" (UID: \"a5ad225a-4ed3-43ba-8cdc-9d4aa1179cb2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7544b6849d-pd9zz" Jul 08 18:46:50.368317 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.368066 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce0fcade-7b07-4a9a-b843-9937f2de7038-registry-tls\") pod \"image-registry-984989766-ktxtr\" (UID: \"ce0fcade-7b07-4a9a-b843-9937f2de7038\") " pod="openshift-image-registry/image-registry-984989766-ktxtr" Jul 08 18:46:50.368317 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.368123 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ce0fcade-7b07-4a9a-b843-9937f2de7038-installation-pull-secrets\") pod \"image-registry-984989766-ktxtr\" (UID: \"ce0fcade-7b07-4a9a-b843-9937f2de7038\") " pod="openshift-image-registry/image-registry-984989766-ktxtr" Jul 08 18:46:50.368317 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.368168 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ce0fcade-7b07-4a9a-b843-9937f2de7038-registry-certificates\") pod \"image-registry-984989766-ktxtr\" (UID: \"ce0fcade-7b07-4a9a-b843-9937f2de7038\") " pod="openshift-image-registry/image-registry-984989766-ktxtr" Jul 08 18:46:50.368317 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.368188 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ce0fcade-7b07-4a9a-b843-9937f2de7038-image-registry-private-configuration\") pod \"image-registry-984989766-ktxtr\" (UID: \"ce0fcade-7b07-4a9a-b843-9937f2de7038\") " pod="openshift-image-registry/image-registry-984989766-ktxtr" Jul 08 18:46:50.368317 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.368219 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce0fcade-7b07-4a9a-b843-9937f2de7038-bound-sa-token\") pod \"image-registry-984989766-ktxtr\" (UID: \"ce0fcade-7b07-4a9a-b843-9937f2de7038\") " pod="openshift-image-registry/image-registry-984989766-ktxtr" Jul 08 18:46:50.368317 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.368269 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzbh4\" (UniqueName: \"kubernetes.io/projected/1b8f10ac-84f9-4e89-91ac-08dbc619b666-kube-api-access-xzbh4\") pod \"console-5db8cf5545-hzsdv\" (UID: \"1b8f10ac-84f9-4e89-91ac-08dbc619b666\") " pod="openshift-console/console-5db8cf5545-hzsdv" Jul 08 18:46:50.368603 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.368320 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a5ad225a-4ed3-43ba-8cdc-9d4aa1179cb2-klusterlet-config\") pod \"klusterlet-addon-workmgr-7544b6849d-pd9zz\" (UID: \"a5ad225a-4ed3-43ba-8cdc-9d4aa1179cb2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7544b6849d-pd9zz" Jul 08 18:46:50.371083 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.371063 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a5ad225a-4ed3-43ba-8cdc-9d4aa1179cb2-klusterlet-config\") pod \"klusterlet-addon-workmgr-7544b6849d-pd9zz\" (UID: \"a5ad225a-4ed3-43ba-8cdc-9d4aa1179cb2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7544b6849d-pd9zz" Jul 08 18:46:50.377938 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.377914 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tcrv\" (UniqueName: \"kubernetes.io/projected/a5ad225a-4ed3-43ba-8cdc-9d4aa1179cb2-kube-api-access-6tcrv\") pod \"klusterlet-addon-workmgr-7544b6849d-pd9zz\" (UID: \"a5ad225a-4ed3-43ba-8cdc-9d4aa1179cb2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7544b6849d-pd9zz" Jul 08 18:46:50.464024 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.463954 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7544b6849d-pd9zz" Jul 08 18:46:50.469028 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.469005 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b8f10ac-84f9-4e89-91ac-08dbc619b666-console-config\") pod \"console-5db8cf5545-hzsdv\" (UID: \"1b8f10ac-84f9-4e89-91ac-08dbc619b666\") " pod="openshift-console/console-5db8cf5545-hzsdv" Jul 08 18:46:50.469121 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.469038 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwk78\" (UniqueName: \"kubernetes.io/projected/ce0fcade-7b07-4a9a-b843-9937f2de7038-kube-api-access-kwk78\") pod \"image-registry-984989766-ktxtr\" (UID: \"ce0fcade-7b07-4a9a-b843-9937f2de7038\") " pod="openshift-image-registry/image-registry-984989766-ktxtr" Jul 08 18:46:50.469121 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.469061 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b8f10ac-84f9-4e89-91ac-08dbc619b666-oauth-serving-cert\") pod \"console-5db8cf5545-hzsdv\" (UID: \"1b8f10ac-84f9-4e89-91ac-08dbc619b666\") " pod="openshift-console/console-5db8cf5545-hzsdv" Jul 08 18:46:50.469198 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.469160 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce0fcade-7b07-4a9a-b843-9937f2de7038-trusted-ca\") pod \"image-registry-984989766-ktxtr\" (UID: \"ce0fcade-7b07-4a9a-b843-9937f2de7038\") " pod="openshift-image-registry/image-registry-984989766-ktxtr" Jul 08 18:46:50.469198 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.469191 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfps9\" (UniqueName: \"kubernetes.io/projected/d11e5bed-ffc0-483d-ad13-435bc9131d71-kube-api-access-bfps9\") pod \"downloads-685bfb5f64-fx4jm\" (UID: \"d11e5bed-ffc0-483d-ad13-435bc9131d71\") " pod="openshift-console/downloads-685bfb5f64-fx4jm" Jul 08 18:46:50.469318 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.469217 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b8f10ac-84f9-4e89-91ac-08dbc619b666-service-ca\") pod \"console-5db8cf5545-hzsdv\" (UID: \"1b8f10ac-84f9-4e89-91ac-08dbc619b666\") " pod="openshift-console/console-5db8cf5545-hzsdv" Jul 08 18:46:50.469318 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.469242 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ce0fcade-7b07-4a9a-b843-9937f2de7038-ca-trust-extracted\") pod \"image-registry-984989766-ktxtr\" (UID: \"ce0fcade-7b07-4a9a-b843-9937f2de7038\") " pod="openshift-image-registry/image-registry-984989766-ktxtr" Jul 08 18:46:50.470147 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.469437 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b8f10ac-84f9-4e89-91ac-08dbc619b666-console-serving-cert\") pod \"console-5db8cf5545-hzsdv\" (UID: \"1b8f10ac-84f9-4e89-91ac-08dbc619b666\") " pod="openshift-console/console-5db8cf5545-hzsdv" Jul 08 18:46:50.470147 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.469492 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b8f10ac-84f9-4e89-91ac-08dbc619b666-console-oauth-config\") pod \"console-5db8cf5545-hzsdv\" (UID: \"1b8f10ac-84f9-4e89-91ac-08dbc619b666\") " pod="openshift-console/console-5db8cf5545-hzsdv" Jul 08 18:46:50.470147 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.469519 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce0fcade-7b07-4a9a-b843-9937f2de7038-registry-tls\") pod \"image-registry-984989766-ktxtr\" (UID: \"ce0fcade-7b07-4a9a-b843-9937f2de7038\") " pod="openshift-image-registry/image-registry-984989766-ktxtr" Jul 08 18:46:50.470147 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.469567 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ce0fcade-7b07-4a9a-b843-9937f2de7038-installation-pull-secrets\") pod \"image-registry-984989766-ktxtr\" (UID: \"ce0fcade-7b07-4a9a-b843-9937f2de7038\") " pod="openshift-image-registry/image-registry-984989766-ktxtr" Jul 08 18:46:50.470147 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.469614 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ce0fcade-7b07-4a9a-b843-9937f2de7038-registry-certificates\") pod \"image-registry-984989766-ktxtr\" (UID: \"ce0fcade-7b07-4a9a-b843-9937f2de7038\") " pod="openshift-image-registry/image-registry-984989766-ktxtr" Jul 08 18:46:50.470147 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.469624 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ce0fcade-7b07-4a9a-b843-9937f2de7038-ca-trust-extracted\") pod \"image-registry-984989766-ktxtr\" (UID: \"ce0fcade-7b07-4a9a-b843-9937f2de7038\") " pod="openshift-image-registry/image-registry-984989766-ktxtr" Jul 08 18:46:50.470147 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.469645 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ce0fcade-7b07-4a9a-b843-9937f2de7038-image-registry-private-configuration\") pod \"image-registry-984989766-ktxtr\" (UID: \"ce0fcade-7b07-4a9a-b843-9937f2de7038\") " pod="openshift-image-registry/image-registry-984989766-ktxtr" Jul 08 18:46:50.470147 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.469672 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce0fcade-7b07-4a9a-b843-9937f2de7038-bound-sa-token\") pod \"image-registry-984989766-ktxtr\" (UID: \"ce0fcade-7b07-4a9a-b843-9937f2de7038\") " pod="openshift-image-registry/image-registry-984989766-ktxtr" Jul 08 18:46:50.470147 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.469705 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzbh4\" (UniqueName: \"kubernetes.io/projected/1b8f10ac-84f9-4e89-91ac-08dbc619b666-kube-api-access-xzbh4\") pod \"console-5db8cf5545-hzsdv\" (UID: \"1b8f10ac-84f9-4e89-91ac-08dbc619b666\") " pod="openshift-console/console-5db8cf5545-hzsdv" Jul 08 18:46:50.470147 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.469756 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b8f10ac-84f9-4e89-91ac-08dbc619b666-console-config\") pod \"console-5db8cf5545-hzsdv\" (UID: \"1b8f10ac-84f9-4e89-91ac-08dbc619b666\") " pod="openshift-console/console-5db8cf5545-hzsdv" Jul 08 18:46:50.470147 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.469797 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b8f10ac-84f9-4e89-91ac-08dbc619b666-oauth-serving-cert\") pod \"console-5db8cf5545-hzsdv\" (UID: \"1b8f10ac-84f9-4e89-91ac-08dbc619b666\") " pod="openshift-console/console-5db8cf5545-hzsdv" Jul 08 18:46:50.470147 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.470001 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b8f10ac-84f9-4e89-91ac-08dbc619b666-service-ca\") pod \"console-5db8cf5545-hzsdv\" (UID: \"1b8f10ac-84f9-4e89-91ac-08dbc619b666\") " pod="openshift-console/console-5db8cf5545-hzsdv" Jul 08 18:46:50.470846 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.470269 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce0fcade-7b07-4a9a-b843-9937f2de7038-trusted-ca\") pod \"image-registry-984989766-ktxtr\" (UID: \"ce0fcade-7b07-4a9a-b843-9937f2de7038\") " pod="openshift-image-registry/image-registry-984989766-ktxtr" Jul 08 18:46:50.470846 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.470522 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ce0fcade-7b07-4a9a-b843-9937f2de7038-registry-certificates\") pod \"image-registry-984989766-ktxtr\" (UID: \"ce0fcade-7b07-4a9a-b843-9937f2de7038\") " pod="openshift-image-registry/image-registry-984989766-ktxtr" Jul 08 18:46:50.472282 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.472244 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce0fcade-7b07-4a9a-b843-9937f2de7038-registry-tls\") pod \"image-registry-984989766-ktxtr\" (UID: \"ce0fcade-7b07-4a9a-b843-9937f2de7038\") " pod="openshift-image-registry/image-registry-984989766-ktxtr" Jul 08 18:46:50.472654 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.472627 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b8f10ac-84f9-4e89-91ac-08dbc619b666-console-oauth-config\") pod \"console-5db8cf5545-hzsdv\" (UID: \"1b8f10ac-84f9-4e89-91ac-08dbc619b666\") " pod="openshift-console/console-5db8cf5545-hzsdv" Jul 08 18:46:50.472809 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.472789 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ce0fcade-7b07-4a9a-b843-9937f2de7038-installation-pull-secrets\") pod \"image-registry-984989766-ktxtr\" (UID: \"ce0fcade-7b07-4a9a-b843-9937f2de7038\") " pod="openshift-image-registry/image-registry-984989766-ktxtr" Jul 08 18:46:50.472920 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.472901 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ce0fcade-7b07-4a9a-b843-9937f2de7038-image-registry-private-configuration\") pod \"image-registry-984989766-ktxtr\" (UID: \"ce0fcade-7b07-4a9a-b843-9937f2de7038\") " pod="openshift-image-registry/image-registry-984989766-ktxtr" Jul 08 18:46:50.472958 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.472922 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b8f10ac-84f9-4e89-91ac-08dbc619b666-console-serving-cert\") pod \"console-5db8cf5545-hzsdv\" (UID: \"1b8f10ac-84f9-4e89-91ac-08dbc619b666\") " pod="openshift-console/console-5db8cf5545-hzsdv" Jul 08 18:46:50.485119 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.485097 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwk78\" (UniqueName: \"kubernetes.io/projected/ce0fcade-7b07-4a9a-b843-9937f2de7038-kube-api-access-kwk78\") pod \"image-registry-984989766-ktxtr\" (UID: \"ce0fcade-7b07-4a9a-b843-9937f2de7038\") " pod="openshift-image-registry/image-registry-984989766-ktxtr" Jul 08 18:46:50.489718 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.489691 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce0fcade-7b07-4a9a-b843-9937f2de7038-bound-sa-token\") pod \"image-registry-984989766-ktxtr\" (UID: \"ce0fcade-7b07-4a9a-b843-9937f2de7038\") " pod="openshift-image-registry/image-registry-984989766-ktxtr" Jul 08 18:46:50.494614 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.494585 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzbh4\" (UniqueName: \"kubernetes.io/projected/1b8f10ac-84f9-4e89-91ac-08dbc619b666-kube-api-access-xzbh4\") pod \"console-5db8cf5545-hzsdv\" (UID: \"1b8f10ac-84f9-4e89-91ac-08dbc619b666\") " pod="openshift-console/console-5db8cf5545-hzsdv" Jul 08 18:46:50.498568 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.498546 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfps9\" (UniqueName: \"kubernetes.io/projected/d11e5bed-ffc0-483d-ad13-435bc9131d71-kube-api-access-bfps9\") pod \"downloads-685bfb5f64-fx4jm\" (UID: \"d11e5bed-ffc0-483d-ad13-435bc9131d71\") " pod="openshift-console/downloads-685bfb5f64-fx4jm" Jul 08 18:46:50.557317 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.557287 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5db8cf5545-hzsdv" Jul 08 18:46:50.565626 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.565604 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-984989766-ktxtr" Jul 08 18:46:50.571853 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:50.571820 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-fx4jm" Jul 08 18:46:51.464725 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:51.464663 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-kxhqr"] Jul 08 18:46:51.473521 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:51.473499 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-kxhqr" Jul 08 18:46:51.480528 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:51.480053 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-kxhqr"] Jul 08 18:46:51.480528 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:51.480293 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-c8r6t\"" Jul 08 18:46:51.487331 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:51.485577 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 08 18:46:51.584975 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:51.583841 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/fccc84da-2d6a-4559-bffb-d0f5958e8471-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-kxhqr\" (UID: \"fccc84da-2d6a-4559-bffb-d0f5958e8471\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-kxhqr" Jul 08 18:46:51.656061 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:51.656026 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-fx4jm"] Jul 08 18:46:51.674405 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:51.672425 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5db8cf5545-hzsdv"] Jul 08 18:46:51.678688 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:46:51.678608 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b8f10ac_84f9_4e89_91ac_08dbc619b666.slice/crio-142d78bb8fff98a02d73f1fa71d6316d0d554d77281a3995c482ceb05ddf45b8 WatchSource:0}: Error finding container 142d78bb8fff98a02d73f1fa71d6316d0d554d77281a3995c482ceb05ddf45b8: Status 404 returned error can't find the container with id 142d78bb8fff98a02d73f1fa71d6316d0d554d77281a3995c482ceb05ddf45b8 Jul 08 18:46:51.686706 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:51.686624 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/fccc84da-2d6a-4559-bffb-d0f5958e8471-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-kxhqr\" (UID: \"fccc84da-2d6a-4559-bffb-d0f5958e8471\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-kxhqr" Jul 08 18:46:51.686818 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:51.686755 2570 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 08 18:46:51.687148 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:51.686868 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fccc84da-2d6a-4559-bffb-d0f5958e8471-tls-certificates podName:fccc84da-2d6a-4559-bffb-d0f5958e8471 nodeName:}" failed. No retries permitted until 2026-07-08 18:46:52.186844602 +0000 UTC m=+73.034119263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/fccc84da-2d6a-4559-bffb-d0f5958e8471-tls-certificates") pod "prometheus-operator-admission-webhook-d559d66d5-kxhqr" (UID: "fccc84da-2d6a-4559-bffb-d0f5958e8471") : secret "prometheus-operator-admission-webhook-tls" not found Jul 08 18:46:51.695670 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:51.695598 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7544b6849d-pd9zz"] Jul 08 18:46:51.717835 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:51.717767 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-984989766-ktxtr"] Jul 08 18:46:51.984808 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:51.984724 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rhwrh" event={"ID":"5f0490c6-dc1a-4e6e-8e53-e9a0eb220119","Type":"ContainerStarted","Data":"42ba65a2d6f3d3fe41feba6f4970953385d7f917c8aac05a92b50bf5586156ad"} Jul 08 18:46:51.984808 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:51.984780 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rhwrh" event={"ID":"5f0490c6-dc1a-4e6e-8e53-e9a0eb220119","Type":"ContainerStarted","Data":"516872e9384d344781af87ceaa9848ffc627d19e76b09ea84f685eddd9157136"} Jul 08 18:46:51.985029 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:51.984849 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rhwrh" Jul 08 18:46:51.985932 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:51.985909 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-fx4jm" event={"ID":"d11e5bed-ffc0-483d-ad13-435bc9131d71","Type":"ContainerStarted","Data":"3e7a3f9c218881dd54f9e7d0f7bd208178b9e24591b3182c6ae461d73ecdfb32"} Jul 08 18:46:51.987614 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:51.987595 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-984989766-ktxtr" event={"ID":"ce0fcade-7b07-4a9a-b843-9937f2de7038","Type":"ContainerStarted","Data":"26c6511155d159a4eae6b6747deb504d18fcf2d98a1f3025b8c3c16da2d33582"} Jul 08 18:46:51.987715 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:51.987629 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-984989766-ktxtr" event={"ID":"ce0fcade-7b07-4a9a-b843-9937f2de7038","Type":"ContainerStarted","Data":"a95c5092991c914a90485c8b774544d33f98bcfa45400a61c9062a49e2464e6e"} Jul 08 18:46:51.987715 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:51.987651 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-984989766-ktxtr" Jul 08 18:46:51.988986 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:51.988961 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5db8cf5545-hzsdv" event={"ID":"1b8f10ac-84f9-4e89-91ac-08dbc619b666","Type":"ContainerStarted","Data":"142d78bb8fff98a02d73f1fa71d6316d0d554d77281a3995c482ceb05ddf45b8"} Jul 08 18:46:51.990582 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:51.990551 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mmgzx" event={"ID":"f7e42624-526e-4aaf-904f-3541dc59aeda","Type":"ContainerStarted","Data":"c8d890efc8c30a43d2a2236641620d6681e78f628a4bd76bb662dc367ddd1096"} Jul 08 18:46:51.990660 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:51.990600 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mmgzx" Jul 08 18:46:51.991964 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:51.991940 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9gjk9" event={"ID":"02e604f5-7b5c-4aec-a685-e380d4b20a3e","Type":"ContainerStarted","Data":"b1fc5792a31332768fcd61fd2774e32d573c1b2ca51fbcdf957f08dca1b12804"} Jul 08 18:46:51.992994 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:51.992975 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7544b6849d-pd9zz" event={"ID":"a5ad225a-4ed3-43ba-8cdc-9d4aa1179cb2","Type":"ContainerStarted","Data":"d4af9efe36e6d964361ac18537f9c10cefdbe1b45306605d4bc6f13ca9f02dbc"} Jul 08 18:46:52.010138 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:52.010101 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rhwrh" podStartSLOduration=33.468956467 podStartE2EDuration="38.010089481s" totalImagesPullingTime="4.541133014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:46:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:46:52.00894324 +0000 UTC m=+72.856217904" watchObservedRunningTime="2026-07-08 18:46:52.010089481 +0000 UTC m=+72.857364146" Jul 08 18:46:52.032953 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:52.032914 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-984989766-ktxtr" podStartSLOduration=2.032900162 podStartE2EDuration="2.032900162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:46:52.032662542 +0000 UTC m=+72.879937209" watchObservedRunningTime="2026-07-08 18:46:52.032900162 +0000 UTC m=+72.880174830" Jul 08 18:46:52.051366 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:52.051327 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mmgzx" podStartSLOduration=67.456369855 podStartE2EDuration="1m13.051317282s" totalImagesPullingTime="5.594947427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:46:52.050852359 +0000 UTC m=+72.898127026" watchObservedRunningTime="2026-07-08 18:46:52.051317282 +0000 UTC m=+72.898591947" Jul 08 18:46:52.075166 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:52.075127 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9gjk9" podStartSLOduration=67.466310541 podStartE2EDuration="1m13.075117585s" totalImagesPullingTime="5.608807044s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:46:52.074237254 +0000 UTC m=+72.921511920" watchObservedRunningTime="2026-07-08 18:46:52.075117585 +0000 UTC m=+72.922392247" Jul 08 18:46:52.190636 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:52.190606 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/fccc84da-2d6a-4559-bffb-d0f5958e8471-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-kxhqr\" (UID: \"fccc84da-2d6a-4559-bffb-d0f5958e8471\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-kxhqr" Jul 08 18:46:52.193352 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:52.193320 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/fccc84da-2d6a-4559-bffb-d0f5958e8471-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-kxhqr\" (UID: \"fccc84da-2d6a-4559-bffb-d0f5958e8471\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-kxhqr" Jul 08 18:46:52.415473 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:52.415439 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-kxhqr" Jul 08 18:46:52.558406 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:52.558087 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-kxhqr"] Jul 08 18:46:52.561664 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:46:52.561628 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfccc84da_2d6a_4559_bffb_d0f5958e8471.slice/crio-d3a8851d397b4696dfc710b98a045fbb2d8299b40a71a8110452699436070820 WatchSource:0}: Error finding container d3a8851d397b4696dfc710b98a045fbb2d8299b40a71a8110452699436070820: Status 404 returned error can't find the container with id d3a8851d397b4696dfc710b98a045fbb2d8299b40a71a8110452699436070820 Jul 08 18:46:53.004145 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:53.003019 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-kxhqr" event={"ID":"fccc84da-2d6a-4559-bffb-d0f5958e8471","Type":"ContainerStarted","Data":"d3a8851d397b4696dfc710b98a045fbb2d8299b40a71a8110452699436070820"} Jul 08 18:46:54.813727 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:54.813658 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-db4fh\" (UID: \"c59f953f-8ffe-4d96-9bcc-68622e7e73f7\") " pod="openshift-insights/insights-runtime-extractor-db4fh" Jul 08 18:46:54.816946 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:54.816918 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c59f953f-8ffe-4d96-9bcc-68622e7e73f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-db4fh\" (UID: \"c59f953f-8ffe-4d96-9bcc-68622e7e73f7\") " pod="openshift-insights/insights-runtime-extractor-db4fh" Jul 08 18:46:54.978049 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:54.978013 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4nhln\"" Jul 08 18:46:54.986481 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:54.986450 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-db4fh" Jul 08 18:46:57.658017 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:57.657927 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-db4fh"] Jul 08 18:46:57.662112 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:46:57.662076 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc59f953f_8ffe_4d96_9bcc_68622e7e73f7.slice/crio-c1fb05ebae24c4328cab6def6a040bd84e00db98aa67e38dd278637fba2341c1 WatchSource:0}: Error finding container c1fb05ebae24c4328cab6def6a040bd84e00db98aa67e38dd278637fba2341c1: Status 404 returned error can't find the container with id c1fb05ebae24c4328cab6def6a040bd84e00db98aa67e38dd278637fba2341c1 Jul 08 18:46:58.026091 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:58.026055 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-kxhqr" event={"ID":"fccc84da-2d6a-4559-bffb-d0f5958e8471","Type":"ContainerStarted","Data":"0da37a079b3019678c0f8128a8aee43d5cdaa6438cc1d4d18d86fac162a3ff7a"} Jul 08 18:46:58.026358 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:58.026329 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-kxhqr" Jul 08 18:46:58.027993 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:58.027962 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5db8cf5545-hzsdv" event={"ID":"1b8f10ac-84f9-4e89-91ac-08dbc619b666","Type":"ContainerStarted","Data":"83cdc8bdf201e680a60ec4b41e3c3c649fe9907818e4724c05633bc580c19f35"} Jul 08 18:46:58.029895 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:58.029865 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-db4fh" event={"ID":"c59f953f-8ffe-4d96-9bcc-68622e7e73f7","Type":"ContainerStarted","Data":"c1fb05ebae24c4328cab6def6a040bd84e00db98aa67e38dd278637fba2341c1"} Jul 08 18:46:58.031826 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:58.031799 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7544b6849d-pd9zz" event={"ID":"a5ad225a-4ed3-43ba-8cdc-9d4aa1179cb2","Type":"ContainerStarted","Data":"4b38d94eb2ee71030bae4ac1f2281dc27c9c58db7bd4cf3f94fc83d421cf0b9e"} Jul 08 18:46:58.032033 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:58.032014 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7544b6849d-pd9zz" Jul 08 18:46:58.032903 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:58.032885 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-kxhqr" Jul 08 18:46:58.033619 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:58.033598 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7544b6849d-pd9zz" Jul 08 18:46:58.043697 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:58.043653 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-kxhqr" podStartSLOduration=2.107144147 podStartE2EDuration="7.043634801s" totalImagesPullingTime="4.936490654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:46:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:46:58.042456164 +0000 UTC m=+78.889730832" watchObservedRunningTime="2026-07-08 18:46:58.043634801 +0000 UTC m=+78.890909467" Jul 08 18:46:58.060980 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:58.060942 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5db8cf5545-hzsdv" podStartSLOduration=2.210537415 podStartE2EDuration="8.060927759s" totalImagesPullingTime="5.850390344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:46:58.059780742 +0000 UTC m=+78.907055409" watchObservedRunningTime="2026-07-08 18:46:58.060927759 +0000 UTC m=+78.908202403" Jul 08 18:46:58.520012 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:58.519952 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7544b6849d-pd9zz" podStartSLOduration=2.673260947 podStartE2EDuration="8.519927966s" totalImagesPullingTime="5.846667019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:46:58.092814459 +0000 UTC m=+78.940089126" watchObservedRunningTime="2026-07-08 18:46:58.519927966 +0000 UTC m=+79.367202633" Jul 08 18:46:58.522870 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:58.521557 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-n588z"] Jul 08 18:46:58.525401 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:58.523967 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-n588z" Jul 08 18:46:58.528164 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:58.527119 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-bnwvm\"" Jul 08 18:46:58.528164 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:58.527357 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 08 18:46:58.528640 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:58.528606 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 08 18:46:58.528924 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:58.528905 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 08 18:46:58.529135 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:58.529107 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 08 18:46:58.529381 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:58.529365 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 08 18:46:58.536166 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:58.536148 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-n588z"] Jul 08 18:46:58.648447 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:58.648412 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l9gx\" (UniqueName: \"kubernetes.io/projected/ca8d865e-4b64-4bda-8030-19555f2cba03-kube-api-access-7l9gx\") pod \"prometheus-operator-567f7b8b7c-n588z\" (UID: \"ca8d865e-4b64-4bda-8030-19555f2cba03\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-n588z" Jul 08 18:46:58.648447 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:58.648461 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca8d865e-4b64-4bda-8030-19555f2cba03-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-n588z\" (UID: \"ca8d865e-4b64-4bda-8030-19555f2cba03\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-n588z" Jul 08 18:46:58.648737 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:58.648500 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca8d865e-4b64-4bda-8030-19555f2cba03-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-n588z\" (UID: \"ca8d865e-4b64-4bda-8030-19555f2cba03\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-n588z" Jul 08 18:46:58.648737 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:58.648570 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca8d865e-4b64-4bda-8030-19555f2cba03-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-n588z\" (UID: \"ca8d865e-4b64-4bda-8030-19555f2cba03\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-n588z" Jul 08 18:46:58.749152 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:58.749113 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca8d865e-4b64-4bda-8030-19555f2cba03-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-n588z\" (UID: \"ca8d865e-4b64-4bda-8030-19555f2cba03\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-n588z" Jul 08 18:46:58.749625 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:58.749183 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l9gx\" (UniqueName: \"kubernetes.io/projected/ca8d865e-4b64-4bda-8030-19555f2cba03-kube-api-access-7l9gx\") pod \"prometheus-operator-567f7b8b7c-n588z\" (UID: \"ca8d865e-4b64-4bda-8030-19555f2cba03\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-n588z" Jul 08 18:46:58.749625 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:58.749215 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca8d865e-4b64-4bda-8030-19555f2cba03-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-n588z\" (UID: \"ca8d865e-4b64-4bda-8030-19555f2cba03\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-n588z" Jul 08 18:46:58.749625 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:58.749265 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca8d865e-4b64-4bda-8030-19555f2cba03-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-n588z\" (UID: \"ca8d865e-4b64-4bda-8030-19555f2cba03\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-n588z" Jul 08 18:46:58.749625 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:58.749375 2570 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 08 18:46:58.749625 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:46:58.749441 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca8d865e-4b64-4bda-8030-19555f2cba03-prometheus-operator-tls podName:ca8d865e-4b64-4bda-8030-19555f2cba03 nodeName:}" failed. No retries permitted until 2026-07-08 18:46:59.249419656 +0000 UTC m=+80.096694306 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/ca8d865e-4b64-4bda-8030-19555f2cba03-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-n588z" (UID: "ca8d865e-4b64-4bda-8030-19555f2cba03") : secret "prometheus-operator-tls" not found Jul 08 18:46:58.750405 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:58.750269 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca8d865e-4b64-4bda-8030-19555f2cba03-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-n588z\" (UID: \"ca8d865e-4b64-4bda-8030-19555f2cba03\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-n588z" Jul 08 18:46:58.754630 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:58.754600 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca8d865e-4b64-4bda-8030-19555f2cba03-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-n588z\" (UID: \"ca8d865e-4b64-4bda-8030-19555f2cba03\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-n588z" Jul 08 18:46:58.763793 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:58.763751 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l9gx\" (UniqueName: \"kubernetes.io/projected/ca8d865e-4b64-4bda-8030-19555f2cba03-kube-api-access-7l9gx\") pod \"prometheus-operator-567f7b8b7c-n588z\" (UID: \"ca8d865e-4b64-4bda-8030-19555f2cba03\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-n588z" Jul 08 18:46:59.253911 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:59.253874 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca8d865e-4b64-4bda-8030-19555f2cba03-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-n588z\" (UID: \"ca8d865e-4b64-4bda-8030-19555f2cba03\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-n588z" Jul 08 18:46:59.256939 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:59.256910 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca8d865e-4b64-4bda-8030-19555f2cba03-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-n588z\" (UID: \"ca8d865e-4b64-4bda-8030-19555f2cba03\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-n588z" Jul 08 18:46:59.440508 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:46:59.440437 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-n588z" Jul 08 18:47:00.558564 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:00.558418 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5db8cf5545-hzsdv" Jul 08 18:47:00.558564 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:00.558495 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5db8cf5545-hzsdv" Jul 08 18:47:00.559787 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:00.559757 2570 patch_prober.go:28] interesting pod/console-5db8cf5545-hzsdv container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.13:8443/health\": dial tcp 10.133.0.13:8443: connect: connection refused" start-of-body= Jul 08 18:47:00.559917 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:00.559809 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-5db8cf5545-hzsdv" podUID="1b8f10ac-84f9-4e89-91ac-08dbc619b666" containerName="console" probeResult="failure" output="Get \"https://10.133.0.13:8443/health\": dial tcp 10.133.0.13:8443: connect: connection refused" Jul 08 18:47:02.005458 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:02.005380 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rhwrh" Jul 08 18:47:02.114603 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:02.114531 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-n588z"] Jul 08 18:47:02.118488 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:47:02.118446 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca8d865e_4b64_4bda_8030_19555f2cba03.slice/crio-128f8c846769d5f2f3886c6fe09c345641d7b8a1882f031dc0a84525233e91b8 WatchSource:0}: Error finding container 128f8c846769d5f2f3886c6fe09c345641d7b8a1882f031dc0a84525233e91b8: Status 404 returned error can't find the container with id 128f8c846769d5f2f3886c6fe09c345641d7b8a1882f031dc0a84525233e91b8 Jul 08 18:47:03.055429 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:03.055338 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-db4fh" event={"ID":"c59f953f-8ffe-4d96-9bcc-68622e7e73f7","Type":"ContainerStarted","Data":"9462606c156dddc9cb95b0535963b62cb0e608b811b617068c3c70a0a64a33b1"} Jul 08 18:47:03.055429 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:03.055425 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-db4fh" event={"ID":"c59f953f-8ffe-4d96-9bcc-68622e7e73f7","Type":"ContainerStarted","Data":"f1d9e13e835cef0a73ae1edbf7bc377aa43dd2045882f0d51b71330448131214"} Jul 08 18:47:03.056683 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:03.056659 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-n588z" event={"ID":"ca8d865e-4b64-4bda-8030-19555f2cba03","Type":"ContainerStarted","Data":"128f8c846769d5f2f3886c6fe09c345641d7b8a1882f031dc0a84525233e91b8"} Jul 08 18:47:10.180623 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:10.180596 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-66d5c86666-wqq74" Jul 08 18:47:10.562975 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:10.562907 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5db8cf5545-hzsdv" Jul 08 18:47:10.568012 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:10.567987 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5db8cf5545-hzsdv" Jul 08 18:47:11.088722 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:11.088682 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-n588z" event={"ID":"ca8d865e-4b64-4bda-8030-19555f2cba03","Type":"ContainerStarted","Data":"070b5c5b7328710cc13f1967bb3aeb175661cc08564da086c2596e5f2d8bf05d"} Jul 08 18:47:11.088880 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:11.088784 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-n588z" event={"ID":"ca8d865e-4b64-4bda-8030-19555f2cba03","Type":"ContainerStarted","Data":"506c915b820c452199520e88a2a5291a58fa1520540eab936572a58e3a0a2876"} Jul 08 18:47:11.091325 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:11.091294 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-fx4jm" event={"ID":"d11e5bed-ffc0-483d-ad13-435bc9131d71","Type":"ContainerStarted","Data":"163f10250852932980519b3e8e50b265f0a79b288d697f24339adabc0fc14448"} Jul 08 18:47:11.091561 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:11.091531 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-fx4jm" Jul 08 18:47:11.093406 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:11.093378 2570 patch_prober.go:28] interesting pod/downloads-685bfb5f64-fx4jm container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.15:8080/\": dial tcp 10.133.0.15:8080: connect: connection refused" start-of-body= Jul 08 18:47:11.093815 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:11.093433 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-685bfb5f64-fx4jm" podUID="d11e5bed-ffc0-483d-ad13-435bc9131d71" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.15:8080/\": dial tcp 10.133.0.15:8080: connect: connection refused" Jul 08 18:47:11.099627 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:11.099602 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5db8cf5545-hzsdv"] Jul 08 18:47:11.113803 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:11.113624 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-n588z" podStartSLOduration=4.346469041 podStartE2EDuration="13.113604811s" totalImagesPullingTime="8.76713577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:47:11.113165889 +0000 UTC m=+91.960440570" watchObservedRunningTime="2026-07-08 18:47:11.113604811 +0000 UTC m=+91.960879478" Jul 08 18:47:11.134608 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:11.134555 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-fx4jm" podStartSLOduration=1.88273813 podStartE2EDuration="21.134537449s" totalImagesPullingTime="19.251799319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:47:11.133478029 +0000 UTC m=+91.980752694" watchObservedRunningTime="2026-07-08 18:47:11.134537449 +0000 UTC m=+91.981812115" Jul 08 18:47:12.107300 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:12.107200 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-fx4jm" Jul 08 18:47:12.903695 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:12.902288 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ghhs2"] Jul 08 18:47:13.024315 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:13.023318 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-984989766-ktxtr" Jul 08 18:47:13.024315 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:13.023367 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ghhs2" Jul 08 18:47:13.027963 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:13.027402 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-q7zvb\"" Jul 08 18:47:13.027963 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:13.027614 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 08 18:47:13.027963 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:13.027812 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 08 18:47:13.030706 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:13.028487 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 08 18:47:13.177529 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:13.177427 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/14933c7c-f7ba-40d9-8ede-f7bd241a9a38-node-exporter-accelerators-collector-config\") pod \"node-exporter-ghhs2\" (UID: \"14933c7c-f7ba-40d9-8ede-f7bd241a9a38\") " pod="openshift-monitoring/node-exporter-ghhs2" Jul 08 18:47:13.177529 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:13.177508 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv6wq\" (UniqueName: \"kubernetes.io/projected/14933c7c-f7ba-40d9-8ede-f7bd241a9a38-kube-api-access-gv6wq\") pod \"node-exporter-ghhs2\" (UID: \"14933c7c-f7ba-40d9-8ede-f7bd241a9a38\") " pod="openshift-monitoring/node-exporter-ghhs2" Jul 08 18:47:13.178094 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:13.177559 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/14933c7c-f7ba-40d9-8ede-f7bd241a9a38-node-exporter-textfile\") pod \"node-exporter-ghhs2\" (UID: \"14933c7c-f7ba-40d9-8ede-f7bd241a9a38\") " pod="openshift-monitoring/node-exporter-ghhs2" Jul 08 18:47:13.178094 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:13.177590 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/14933c7c-f7ba-40d9-8ede-f7bd241a9a38-node-exporter-wtmp\") pod \"node-exporter-ghhs2\" (UID: \"14933c7c-f7ba-40d9-8ede-f7bd241a9a38\") " pod="openshift-monitoring/node-exporter-ghhs2" Jul 08 18:47:13.178094 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:13.177617 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14933c7c-f7ba-40d9-8ede-f7bd241a9a38-node-exporter-tls\") pod \"node-exporter-ghhs2\" (UID: \"14933c7c-f7ba-40d9-8ede-f7bd241a9a38\") " pod="openshift-monitoring/node-exporter-ghhs2" Jul 08 18:47:13.178094 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:13.177666 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14933c7c-f7ba-40d9-8ede-f7bd241a9a38-metrics-client-ca\") pod \"node-exporter-ghhs2\" (UID: \"14933c7c-f7ba-40d9-8ede-f7bd241a9a38\") " pod="openshift-monitoring/node-exporter-ghhs2" Jul 08 18:47:13.178094 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:13.177724 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/14933c7c-f7ba-40d9-8ede-f7bd241a9a38-root\") pod \"node-exporter-ghhs2\" (UID: \"14933c7c-f7ba-40d9-8ede-f7bd241a9a38\") " pod="openshift-monitoring/node-exporter-ghhs2" Jul 08 18:47:13.178094 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:13.177774 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14933c7c-f7ba-40d9-8ede-f7bd241a9a38-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ghhs2\" (UID: \"14933c7c-f7ba-40d9-8ede-f7bd241a9a38\") " pod="openshift-monitoring/node-exporter-ghhs2" Jul 08 18:47:13.178094 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:13.177835 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14933c7c-f7ba-40d9-8ede-f7bd241a9a38-sys\") pod \"node-exporter-ghhs2\" (UID: \"14933c7c-f7ba-40d9-8ede-f7bd241a9a38\") " pod="openshift-monitoring/node-exporter-ghhs2" Jul 08 18:47:13.278589 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:13.278553 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14933c7c-f7ba-40d9-8ede-f7bd241a9a38-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ghhs2\" (UID: \"14933c7c-f7ba-40d9-8ede-f7bd241a9a38\") " pod="openshift-monitoring/node-exporter-ghhs2" Jul 08 18:47:13.278589 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:13.278604 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14933c7c-f7ba-40d9-8ede-f7bd241a9a38-sys\") pod \"node-exporter-ghhs2\" (UID: \"14933c7c-f7ba-40d9-8ede-f7bd241a9a38\") " pod="openshift-monitoring/node-exporter-ghhs2" Jul 08 18:47:13.278832 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:13.278672 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14933c7c-f7ba-40d9-8ede-f7bd241a9a38-sys\") pod \"node-exporter-ghhs2\" (UID: \"14933c7c-f7ba-40d9-8ede-f7bd241a9a38\") " pod="openshift-monitoring/node-exporter-ghhs2" Jul 08 18:47:13.278832 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:13.278710 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/14933c7c-f7ba-40d9-8ede-f7bd241a9a38-node-exporter-accelerators-collector-config\") pod \"node-exporter-ghhs2\" (UID: \"14933c7c-f7ba-40d9-8ede-f7bd241a9a38\") " pod="openshift-monitoring/node-exporter-ghhs2" Jul 08 18:47:13.278832 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:13.278748 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv6wq\" (UniqueName: \"kubernetes.io/projected/14933c7c-f7ba-40d9-8ede-f7bd241a9a38-kube-api-access-gv6wq\") pod \"node-exporter-ghhs2\" (UID: \"14933c7c-f7ba-40d9-8ede-f7bd241a9a38\") " pod="openshift-monitoring/node-exporter-ghhs2" Jul 08 18:47:13.278832 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:13.278786 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/14933c7c-f7ba-40d9-8ede-f7bd241a9a38-node-exporter-textfile\") pod \"node-exporter-ghhs2\" (UID: \"14933c7c-f7ba-40d9-8ede-f7bd241a9a38\") " pod="openshift-monitoring/node-exporter-ghhs2" Jul 08 18:47:13.278832 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:13.278816 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/14933c7c-f7ba-40d9-8ede-f7bd241a9a38-node-exporter-wtmp\") pod \"node-exporter-ghhs2\" (UID: \"14933c7c-f7ba-40d9-8ede-f7bd241a9a38\") " pod="openshift-monitoring/node-exporter-ghhs2" Jul 08 18:47:13.279081 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:13.278841 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14933c7c-f7ba-40d9-8ede-f7bd241a9a38-node-exporter-tls\") pod \"node-exporter-ghhs2\" (UID: \"14933c7c-f7ba-40d9-8ede-f7bd241a9a38\") " pod="openshift-monitoring/node-exporter-ghhs2" Jul 08 18:47:13.279081 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:13.278867 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14933c7c-f7ba-40d9-8ede-f7bd241a9a38-metrics-client-ca\") pod \"node-exporter-ghhs2\" (UID: \"14933c7c-f7ba-40d9-8ede-f7bd241a9a38\") " pod="openshift-monitoring/node-exporter-ghhs2" Jul 08 18:47:13.279081 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:13.278900 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/14933c7c-f7ba-40d9-8ede-f7bd241a9a38-root\") pod \"node-exporter-ghhs2\" (UID: \"14933c7c-f7ba-40d9-8ede-f7bd241a9a38\") " pod="openshift-monitoring/node-exporter-ghhs2" Jul 08 18:47:13.279081 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:13.279006 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/14933c7c-f7ba-40d9-8ede-f7bd241a9a38-node-exporter-wtmp\") pod \"node-exporter-ghhs2\" (UID: \"14933c7c-f7ba-40d9-8ede-f7bd241a9a38\") " pod="openshift-monitoring/node-exporter-ghhs2" Jul 08 18:47:13.279297 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:13.279108 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/14933c7c-f7ba-40d9-8ede-f7bd241a9a38-root\") pod \"node-exporter-ghhs2\" (UID: \"14933c7c-f7ba-40d9-8ede-f7bd241a9a38\") " pod="openshift-monitoring/node-exporter-ghhs2" Jul 08 18:47:13.279297 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:13.279201 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/14933c7c-f7ba-40d9-8ede-f7bd241a9a38-node-exporter-textfile\") pod \"node-exporter-ghhs2\" (UID: \"14933c7c-f7ba-40d9-8ede-f7bd241a9a38\") " pod="openshift-monitoring/node-exporter-ghhs2" Jul 08 18:47:13.279840 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:13.279692 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14933c7c-f7ba-40d9-8ede-f7bd241a9a38-metrics-client-ca\") pod \"node-exporter-ghhs2\" (UID: \"14933c7c-f7ba-40d9-8ede-f7bd241a9a38\") " pod="openshift-monitoring/node-exporter-ghhs2" Jul 08 18:47:13.279840 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:13.279718 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/14933c7c-f7ba-40d9-8ede-f7bd241a9a38-node-exporter-accelerators-collector-config\") pod \"node-exporter-ghhs2\" (UID: \"14933c7c-f7ba-40d9-8ede-f7bd241a9a38\") " pod="openshift-monitoring/node-exporter-ghhs2" Jul 08 18:47:13.282132 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:13.281904 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14933c7c-f7ba-40d9-8ede-f7bd241a9a38-node-exporter-tls\") pod \"node-exporter-ghhs2\" (UID: \"14933c7c-f7ba-40d9-8ede-f7bd241a9a38\") " pod="openshift-monitoring/node-exporter-ghhs2" Jul 08 18:47:13.282132 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:13.282071 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14933c7c-f7ba-40d9-8ede-f7bd241a9a38-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ghhs2\" (UID: \"14933c7c-f7ba-40d9-8ede-f7bd241a9a38\") " pod="openshift-monitoring/node-exporter-ghhs2" Jul 08 18:47:13.288604 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:13.288577 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv6wq\" (UniqueName: \"kubernetes.io/projected/14933c7c-f7ba-40d9-8ede-f7bd241a9a38-kube-api-access-gv6wq\") pod \"node-exporter-ghhs2\" (UID: \"14933c7c-f7ba-40d9-8ede-f7bd241a9a38\") " pod="openshift-monitoring/node-exporter-ghhs2" Jul 08 18:47:13.340598 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:13.340569 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ghhs2" Jul 08 18:47:13.384417 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:47:13.384373 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14933c7c_f7ba_40d9_8ede_f7bd241a9a38.slice/crio-a5bb19b72bf681490f72d2c97d7517b0ff1a97da23ddaacbb79b30864fbf92f2 WatchSource:0}: Error finding container a5bb19b72bf681490f72d2c97d7517b0ff1a97da23ddaacbb79b30864fbf92f2: Status 404 returned error can't find the container with id a5bb19b72bf681490f72d2c97d7517b0ff1a97da23ddaacbb79b30864fbf92f2 Jul 08 18:47:14.111043 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:14.111003 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-db4fh" event={"ID":"c59f953f-8ffe-4d96-9bcc-68622e7e73f7","Type":"ContainerStarted","Data":"4da496d6efdd849f4980406c17475530ba5a1ceaec5c7db3feb636a19cc889e1"} Jul 08 18:47:14.113573 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:14.113534 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ghhs2" event={"ID":"14933c7c-f7ba-40d9-8ede-f7bd241a9a38","Type":"ContainerStarted","Data":"a5bb19b72bf681490f72d2c97d7517b0ff1a97da23ddaacbb79b30864fbf92f2"} Jul 08 18:47:14.135357 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:14.135295 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-db4fh" podStartSLOduration=36.15194727 podStartE2EDuration="52.135274419s" totalImagesPullingTime="15.983327149s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:46:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:47:14.13397702 +0000 UTC m=+94.981251692" watchObservedRunningTime="2026-07-08 18:47:14.135274419 +0000 UTC m=+94.982549090" Jul 08 18:47:14.970329 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:14.970294 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7756484b49-bg58n"] Jul 08 18:47:14.990563 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:14.990528 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7756484b49-bg58n"] Jul 08 18:47:14.990736 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:14.990644 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7756484b49-bg58n" Jul 08 18:47:14.994328 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:14.993693 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 08 18:47:14.994328 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:14.993884 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 08 18:47:14.994328 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:14.994187 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 08 18:47:14.994573 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:14.994400 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-v7bpf\"" Jul 08 18:47:14.994573 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:14.994427 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 08 18:47:14.994711 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:14.994675 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 08 18:47:14.994769 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:14.994730 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-1st480tkar9ts\"" Jul 08 18:47:15.094712 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.094681 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a41d4b66-d1a1-4393-83a8-de9bbe36c08a-metrics-client-ca\") pod \"thanos-querier-7756484b49-bg58n\" (UID: \"a41d4b66-d1a1-4393-83a8-de9bbe36c08a\") " pod="openshift-monitoring/thanos-querier-7756484b49-bg58n" Jul 08 18:47:15.094855 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.094756 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a41d4b66-d1a1-4393-83a8-de9bbe36c08a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7756484b49-bg58n\" (UID: \"a41d4b66-d1a1-4393-83a8-de9bbe36c08a\") " pod="openshift-monitoring/thanos-querier-7756484b49-bg58n" Jul 08 18:47:15.094855 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.094804 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a41d4b66-d1a1-4393-83a8-de9bbe36c08a-secret-grpc-tls\") pod \"thanos-querier-7756484b49-bg58n\" (UID: \"a41d4b66-d1a1-4393-83a8-de9bbe36c08a\") " pod="openshift-monitoring/thanos-querier-7756484b49-bg58n" Jul 08 18:47:15.094974 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.094868 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a41d4b66-d1a1-4393-83a8-de9bbe36c08a-secret-thanos-querier-tls\") pod \"thanos-querier-7756484b49-bg58n\" (UID: \"a41d4b66-d1a1-4393-83a8-de9bbe36c08a\") " pod="openshift-monitoring/thanos-querier-7756484b49-bg58n" Jul 08 18:47:15.094974 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.094910 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a41d4b66-d1a1-4393-83a8-de9bbe36c08a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7756484b49-bg58n\" (UID: \"a41d4b66-d1a1-4393-83a8-de9bbe36c08a\") " pod="openshift-monitoring/thanos-querier-7756484b49-bg58n" Jul 08 18:47:15.094974 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.094933 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a41d4b66-d1a1-4393-83a8-de9bbe36c08a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7756484b49-bg58n\" (UID: \"a41d4b66-d1a1-4393-83a8-de9bbe36c08a\") " pod="openshift-monitoring/thanos-querier-7756484b49-bg58n" Jul 08 18:47:15.094974 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.094955 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a41d4b66-d1a1-4393-83a8-de9bbe36c08a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7756484b49-bg58n\" (UID: \"a41d4b66-d1a1-4393-83a8-de9bbe36c08a\") " pod="openshift-monitoring/thanos-querier-7756484b49-bg58n" Jul 08 18:47:15.095156 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.094978 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4vtx\" (UniqueName: \"kubernetes.io/projected/a41d4b66-d1a1-4393-83a8-de9bbe36c08a-kube-api-access-d4vtx\") pod \"thanos-querier-7756484b49-bg58n\" (UID: \"a41d4b66-d1a1-4393-83a8-de9bbe36c08a\") " pod="openshift-monitoring/thanos-querier-7756484b49-bg58n" Jul 08 18:47:15.196480 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.196441 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a41d4b66-d1a1-4393-83a8-de9bbe36c08a-secret-grpc-tls\") pod \"thanos-querier-7756484b49-bg58n\" (UID: \"a41d4b66-d1a1-4393-83a8-de9bbe36c08a\") " pod="openshift-monitoring/thanos-querier-7756484b49-bg58n" Jul 08 18:47:15.196670 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.196487 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a41d4b66-d1a1-4393-83a8-de9bbe36c08a-secret-thanos-querier-tls\") pod \"thanos-querier-7756484b49-bg58n\" (UID: \"a41d4b66-d1a1-4393-83a8-de9bbe36c08a\") " pod="openshift-monitoring/thanos-querier-7756484b49-bg58n" Jul 08 18:47:15.196670 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.196511 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a41d4b66-d1a1-4393-83a8-de9bbe36c08a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7756484b49-bg58n\" (UID: \"a41d4b66-d1a1-4393-83a8-de9bbe36c08a\") " pod="openshift-monitoring/thanos-querier-7756484b49-bg58n" Jul 08 18:47:15.196670 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.196542 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a41d4b66-d1a1-4393-83a8-de9bbe36c08a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7756484b49-bg58n\" (UID: \"a41d4b66-d1a1-4393-83a8-de9bbe36c08a\") " pod="openshift-monitoring/thanos-querier-7756484b49-bg58n" Jul 08 18:47:15.196670 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.196564 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a41d4b66-d1a1-4393-83a8-de9bbe36c08a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7756484b49-bg58n\" (UID: \"a41d4b66-d1a1-4393-83a8-de9bbe36c08a\") " pod="openshift-monitoring/thanos-querier-7756484b49-bg58n" Jul 08 18:47:15.196670 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.196583 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4vtx\" (UniqueName: \"kubernetes.io/projected/a41d4b66-d1a1-4393-83a8-de9bbe36c08a-kube-api-access-d4vtx\") pod \"thanos-querier-7756484b49-bg58n\" (UID: \"a41d4b66-d1a1-4393-83a8-de9bbe36c08a\") " pod="openshift-monitoring/thanos-querier-7756484b49-bg58n" Jul 08 18:47:15.197387 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.197059 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a41d4b66-d1a1-4393-83a8-de9bbe36c08a-metrics-client-ca\") pod \"thanos-querier-7756484b49-bg58n\" (UID: \"a41d4b66-d1a1-4393-83a8-de9bbe36c08a\") " pod="openshift-monitoring/thanos-querier-7756484b49-bg58n" Jul 08 18:47:15.197387 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.197162 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a41d4b66-d1a1-4393-83a8-de9bbe36c08a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7756484b49-bg58n\" (UID: \"a41d4b66-d1a1-4393-83a8-de9bbe36c08a\") " pod="openshift-monitoring/thanos-querier-7756484b49-bg58n" Jul 08 18:47:15.197946 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.197880 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a41d4b66-d1a1-4393-83a8-de9bbe36c08a-metrics-client-ca\") pod \"thanos-querier-7756484b49-bg58n\" (UID: \"a41d4b66-d1a1-4393-83a8-de9bbe36c08a\") " pod="openshift-monitoring/thanos-querier-7756484b49-bg58n" Jul 08 18:47:15.199803 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.199775 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a41d4b66-d1a1-4393-83a8-de9bbe36c08a-secret-thanos-querier-tls\") pod \"thanos-querier-7756484b49-bg58n\" (UID: \"a41d4b66-d1a1-4393-83a8-de9bbe36c08a\") " pod="openshift-monitoring/thanos-querier-7756484b49-bg58n" Jul 08 18:47:15.200010 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.199982 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a41d4b66-d1a1-4393-83a8-de9bbe36c08a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7756484b49-bg58n\" (UID: \"a41d4b66-d1a1-4393-83a8-de9bbe36c08a\") " pod="openshift-monitoring/thanos-querier-7756484b49-bg58n" Jul 08 18:47:15.200641 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.200607 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a41d4b66-d1a1-4393-83a8-de9bbe36c08a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7756484b49-bg58n\" (UID: \"a41d4b66-d1a1-4393-83a8-de9bbe36c08a\") " pod="openshift-monitoring/thanos-querier-7756484b49-bg58n" Jul 08 18:47:15.201136 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.201087 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a41d4b66-d1a1-4393-83a8-de9bbe36c08a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7756484b49-bg58n\" (UID: \"a41d4b66-d1a1-4393-83a8-de9bbe36c08a\") " pod="openshift-monitoring/thanos-querier-7756484b49-bg58n" Jul 08 18:47:15.201270 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.201239 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a41d4b66-d1a1-4393-83a8-de9bbe36c08a-secret-grpc-tls\") pod \"thanos-querier-7756484b49-bg58n\" (UID: \"a41d4b66-d1a1-4393-83a8-de9bbe36c08a\") " pod="openshift-monitoring/thanos-querier-7756484b49-bg58n" Jul 08 18:47:15.201415 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.201370 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a41d4b66-d1a1-4393-83a8-de9bbe36c08a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7756484b49-bg58n\" (UID: \"a41d4b66-d1a1-4393-83a8-de9bbe36c08a\") " pod="openshift-monitoring/thanos-querier-7756484b49-bg58n" Jul 08 18:47:15.203604 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.203526 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-66d5c86666-wqq74" podUID="2e05edf7-1d8f-4a6d-b935-4d2d91447b48" containerName="registry" containerID="cri-o://fdf9546880615a231f4964457e2e5014b4fa4ccf00c2b937bed0d0cbf6bad46b" gracePeriod=30 Jul 08 18:47:15.206636 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.206592 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4vtx\" (UniqueName: \"kubernetes.io/projected/a41d4b66-d1a1-4393-83a8-de9bbe36c08a-kube-api-access-d4vtx\") pod \"thanos-querier-7756484b49-bg58n\" (UID: \"a41d4b66-d1a1-4393-83a8-de9bbe36c08a\") " pod="openshift-monitoring/thanos-querier-7756484b49-bg58n" Jul 08 18:47:15.304039 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.304008 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7756484b49-bg58n" Jul 08 18:47:15.486697 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.486495 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7756484b49-bg58n"] Jul 08 18:47:15.488026 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.487993 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66d5c86666-wqq74" Jul 08 18:47:15.489624 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:47:15.489593 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda41d4b66_d1a1_4393_83a8_de9bbe36c08a.slice/crio-384b1225970220da5341cca5fd9df96187980e3796ecf4f43e6fb2ead12127ff WatchSource:0}: Error finding container 384b1225970220da5341cca5fd9df96187980e3796ecf4f43e6fb2ead12127ff: Status 404 returned error can't find the container with id 384b1225970220da5341cca5fd9df96187980e3796ecf4f43e6fb2ead12127ff Jul 08 18:47:15.600789 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.600746 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-registry-tls\") pod \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " Jul 08 18:47:15.600789 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.600787 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2ccx\" (UniqueName: \"kubernetes.io/projected/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-kube-api-access-v2ccx\") pod \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " Jul 08 18:47:15.601040 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.600819 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-installation-pull-secrets\") pod \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " Jul 08 18:47:15.601040 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.600852 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-bound-sa-token\") pod \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " Jul 08 18:47:15.601040 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.600877 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-image-registry-private-configuration\") pod \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " Jul 08 18:47:15.601040 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.600916 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-registry-certificates\") pod \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " Jul 08 18:47:15.601040 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.600946 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-ca-trust-extracted\") pod \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " Jul 08 18:47:15.601040 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.600991 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-trusted-ca\") pod \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\" (UID: \"2e05edf7-1d8f-4a6d-b935-4d2d91447b48\") " Jul 08 18:47:15.601553 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.601491 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2e05edf7-1d8f-4a6d-b935-4d2d91447b48" (UID: "2e05edf7-1d8f-4a6d-b935-4d2d91447b48"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:47:15.601693 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.601571 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2e05edf7-1d8f-4a6d-b935-4d2d91447b48" (UID: "2e05edf7-1d8f-4a6d-b935-4d2d91447b48"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:47:15.604112 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.604080 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2e05edf7-1d8f-4a6d-b935-4d2d91447b48" (UID: "2e05edf7-1d8f-4a6d-b935-4d2d91447b48"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:47:15.604222 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.604151 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2e05edf7-1d8f-4a6d-b935-4d2d91447b48" (UID: "2e05edf7-1d8f-4a6d-b935-4d2d91447b48"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:47:15.604222 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.604165 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2e05edf7-1d8f-4a6d-b935-4d2d91447b48" (UID: "2e05edf7-1d8f-4a6d-b935-4d2d91447b48"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:47:15.604535 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.604512 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-kube-api-access-v2ccx" (OuterVolumeSpecName: "kube-api-access-v2ccx") pod "2e05edf7-1d8f-4a6d-b935-4d2d91447b48" (UID: "2e05edf7-1d8f-4a6d-b935-4d2d91447b48"). InnerVolumeSpecName "kube-api-access-v2ccx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:47:15.605641 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.605617 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2e05edf7-1d8f-4a6d-b935-4d2d91447b48" (UID: "2e05edf7-1d8f-4a6d-b935-4d2d91447b48"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:47:15.613412 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.613388 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2e05edf7-1d8f-4a6d-b935-4d2d91447b48" (UID: "2e05edf7-1d8f-4a6d-b935-4d2d91447b48"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:47:15.701602 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.701575 2570 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-registry-certificates\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:47:15.701602 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.701603 2570 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-ca-trust-extracted\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:47:15.701776 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.701617 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-trusted-ca\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:47:15.701776 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.701630 2570 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-registry-tls\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:47:15.701776 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.701644 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2ccx\" (UniqueName: \"kubernetes.io/projected/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-kube-api-access-v2ccx\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:47:15.701776 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.701656 2570 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-installation-pull-secrets\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:47:15.701776 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.701669 2570 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-bound-sa-token\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:47:15.701776 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:15.701683 2570 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e05edf7-1d8f-4a6d-b935-4d2d91447b48-image-registry-private-configuration\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:47:16.125818 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:16.125766 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7756484b49-bg58n" event={"ID":"a41d4b66-d1a1-4393-83a8-de9bbe36c08a","Type":"ContainerStarted","Data":"384b1225970220da5341cca5fd9df96187980e3796ecf4f43e6fb2ead12127ff"} Jul 08 18:47:16.129101 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:16.129067 2570 generic.go:356] "Generic (PLEG): container finished" podID="2e05edf7-1d8f-4a6d-b935-4d2d91447b48" containerID="fdf9546880615a231f4964457e2e5014b4fa4ccf00c2b937bed0d0cbf6bad46b" exitCode=0 Jul 08 18:47:16.129816 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:16.129168 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66d5c86666-wqq74" Jul 08 18:47:16.129816 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:16.129185 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66d5c86666-wqq74" event={"ID":"2e05edf7-1d8f-4a6d-b935-4d2d91447b48","Type":"ContainerDied","Data":"fdf9546880615a231f4964457e2e5014b4fa4ccf00c2b937bed0d0cbf6bad46b"} Jul 08 18:47:16.129816 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:16.129472 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66d5c86666-wqq74" event={"ID":"2e05edf7-1d8f-4a6d-b935-4d2d91447b48","Type":"ContainerDied","Data":"e6a1c93fac81df9a589aed6f23e44335968416b50a9b028301e2ec996098e8a0"} Jul 08 18:47:16.129816 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:16.129746 2570 scope.go:117] "RemoveContainer" containerID="fdf9546880615a231f4964457e2e5014b4fa4ccf00c2b937bed0d0cbf6bad46b" Jul 08 18:47:16.136000 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:16.135969 2570 generic.go:356] "Generic (PLEG): container finished" podID="14933c7c-f7ba-40d9-8ede-f7bd241a9a38" containerID="def15b84715e0b7ab02c5680de009c97d9851b28eae3d521bacf8ea4a0643f2d" exitCode=0 Jul 08 18:47:16.136109 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:16.136016 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ghhs2" event={"ID":"14933c7c-f7ba-40d9-8ede-f7bd241a9a38","Type":"ContainerDied","Data":"def15b84715e0b7ab02c5680de009c97d9851b28eae3d521bacf8ea4a0643f2d"} Jul 08 18:47:16.145976 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:16.145888 2570 scope.go:117] "RemoveContainer" containerID="fdf9546880615a231f4964457e2e5014b4fa4ccf00c2b937bed0d0cbf6bad46b" Jul 08 18:47:16.147377 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:47:16.147296 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdf9546880615a231f4964457e2e5014b4fa4ccf00c2b937bed0d0cbf6bad46b\": container with ID starting with fdf9546880615a231f4964457e2e5014b4fa4ccf00c2b937bed0d0cbf6bad46b not found: ID does not exist" containerID="fdf9546880615a231f4964457e2e5014b4fa4ccf00c2b937bed0d0cbf6bad46b" Jul 08 18:47:16.147377 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:16.147335 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdf9546880615a231f4964457e2e5014b4fa4ccf00c2b937bed0d0cbf6bad46b"} err="failed to get container status \"fdf9546880615a231f4964457e2e5014b4fa4ccf00c2b937bed0d0cbf6bad46b\": rpc error: code = NotFound desc = could not find container \"fdf9546880615a231f4964457e2e5014b4fa4ccf00c2b937bed0d0cbf6bad46b\": container with ID starting with fdf9546880615a231f4964457e2e5014b4fa4ccf00c2b937bed0d0cbf6bad46b not found: ID does not exist" Jul 08 18:47:16.149553 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:16.149534 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66d5c86666-wqq74"] Jul 08 18:47:16.154905 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:16.154870 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-66d5c86666-wqq74"] Jul 08 18:47:17.143023 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.142986 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ghhs2" event={"ID":"14933c7c-f7ba-40d9-8ede-f7bd241a9a38","Type":"ContainerStarted","Data":"1932a88dd5e37b4b045ec2ef63df1fc527525128be12462d4ef07a51be7e1d52"} Jul 08 18:47:17.143616 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.143054 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ghhs2" event={"ID":"14933c7c-f7ba-40d9-8ede-f7bd241a9a38","Type":"ContainerStarted","Data":"72e9ff4662a0853984e068e2323337944365b165a794544913a776876a5b2215"} Jul 08 18:47:17.269763 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.269712 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ghhs2" podStartSLOduration=3.529766709 podStartE2EDuration="5.269693564s" totalImagesPullingTime="1.633598115s" totalInitContainerRuntime="106.32874ms" isStatefulPod=true podCreationTimestamp="2026-07-08 18:47:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:47:17.164097799 +0000 UTC m=+98.011372466" watchObservedRunningTime="2026-07-08 18:47:17.269693564 +0000 UTC m=+98.116968229" Jul 08 18:47:17.271121 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.271095 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6bcb9bcc4b-5wrlm"] Jul 08 18:47:17.271515 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.271491 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e05edf7-1d8f-4a6d-b935-4d2d91447b48" containerName="registry" Jul 08 18:47:17.271515 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.271513 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e05edf7-1d8f-4a6d-b935-4d2d91447b48" containerName="registry" Jul 08 18:47:17.271682 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.271560 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e05edf7-1d8f-4a6d-b935-4d2d91447b48" containerName="registry" Jul 08 18:47:17.287660 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.287640 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6bcb9bcc4b-5wrlm"] Jul 08 18:47:17.287769 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.287746 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6bcb9bcc4b-5wrlm" Jul 08 18:47:17.293433 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.293399 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 08 18:47:17.293433 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.293420 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 08 18:47:17.293929 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.293873 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 08 18:47:17.296728 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.295900 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 08 18:47:17.297162 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.296982 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-vm7ff\"" Jul 08 18:47:17.298854 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.298399 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6096tc9ifj2gk\"" Jul 08 18:47:17.416159 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.416080 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9b81ec53-6553-4e90-9013-57f222fad5e6-audit-log\") pod \"metrics-server-6bcb9bcc4b-5wrlm\" (UID: \"9b81ec53-6553-4e90-9013-57f222fad5e6\") " pod="openshift-monitoring/metrics-server-6bcb9bcc4b-5wrlm" Jul 08 18:47:17.416159 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.416127 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4flcz\" (UniqueName: \"kubernetes.io/projected/9b81ec53-6553-4e90-9013-57f222fad5e6-kube-api-access-4flcz\") pod \"metrics-server-6bcb9bcc4b-5wrlm\" (UID: \"9b81ec53-6553-4e90-9013-57f222fad5e6\") " pod="openshift-monitoring/metrics-server-6bcb9bcc4b-5wrlm" Jul 08 18:47:17.416159 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.416154 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9b81ec53-6553-4e90-9013-57f222fad5e6-metrics-server-audit-profiles\") pod \"metrics-server-6bcb9bcc4b-5wrlm\" (UID: \"9b81ec53-6553-4e90-9013-57f222fad5e6\") " pod="openshift-monitoring/metrics-server-6bcb9bcc4b-5wrlm" Jul 08 18:47:17.416451 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.416189 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9b81ec53-6553-4e90-9013-57f222fad5e6-secret-metrics-server-client-certs\") pod \"metrics-server-6bcb9bcc4b-5wrlm\" (UID: \"9b81ec53-6553-4e90-9013-57f222fad5e6\") " pod="openshift-monitoring/metrics-server-6bcb9bcc4b-5wrlm" Jul 08 18:47:17.416451 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.416216 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b81ec53-6553-4e90-9013-57f222fad5e6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6bcb9bcc4b-5wrlm\" (UID: \"9b81ec53-6553-4e90-9013-57f222fad5e6\") " pod="openshift-monitoring/metrics-server-6bcb9bcc4b-5wrlm" Jul 08 18:47:17.416451 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.416240 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9b81ec53-6553-4e90-9013-57f222fad5e6-secret-metrics-server-tls\") pod \"metrics-server-6bcb9bcc4b-5wrlm\" (UID: \"9b81ec53-6553-4e90-9013-57f222fad5e6\") " pod="openshift-monitoring/metrics-server-6bcb9bcc4b-5wrlm" Jul 08 18:47:17.416451 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.416308 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9b81ec53-6553-4e90-9013-57f222fad5e6-client-ca-bundle\") pod \"metrics-server-6bcb9bcc4b-5wrlm\" (UID: \"9b81ec53-6553-4e90-9013-57f222fad5e6\") " pod="openshift-monitoring/metrics-server-6bcb9bcc4b-5wrlm" Jul 08 18:47:17.517264 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.516825 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b81ec53-6553-4e90-9013-57f222fad5e6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6bcb9bcc4b-5wrlm\" (UID: \"9b81ec53-6553-4e90-9013-57f222fad5e6\") " pod="openshift-monitoring/metrics-server-6bcb9bcc4b-5wrlm" Jul 08 18:47:17.517264 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.516885 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9b81ec53-6553-4e90-9013-57f222fad5e6-secret-metrics-server-tls\") pod \"metrics-server-6bcb9bcc4b-5wrlm\" (UID: \"9b81ec53-6553-4e90-9013-57f222fad5e6\") " pod="openshift-monitoring/metrics-server-6bcb9bcc4b-5wrlm" Jul 08 18:47:17.517264 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.516943 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9b81ec53-6553-4e90-9013-57f222fad5e6-client-ca-bundle\") pod \"metrics-server-6bcb9bcc4b-5wrlm\" (UID: \"9b81ec53-6553-4e90-9013-57f222fad5e6\") " pod="openshift-monitoring/metrics-server-6bcb9bcc4b-5wrlm" Jul 08 18:47:17.517264 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.516998 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9b81ec53-6553-4e90-9013-57f222fad5e6-audit-log\") pod \"metrics-server-6bcb9bcc4b-5wrlm\" (UID: \"9b81ec53-6553-4e90-9013-57f222fad5e6\") " pod="openshift-monitoring/metrics-server-6bcb9bcc4b-5wrlm" Jul 08 18:47:17.517264 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.517027 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4flcz\" (UniqueName: \"kubernetes.io/projected/9b81ec53-6553-4e90-9013-57f222fad5e6-kube-api-access-4flcz\") pod \"metrics-server-6bcb9bcc4b-5wrlm\" (UID: \"9b81ec53-6553-4e90-9013-57f222fad5e6\") " pod="openshift-monitoring/metrics-server-6bcb9bcc4b-5wrlm" Jul 08 18:47:17.517264 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.517058 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9b81ec53-6553-4e90-9013-57f222fad5e6-metrics-server-audit-profiles\") pod \"metrics-server-6bcb9bcc4b-5wrlm\" (UID: \"9b81ec53-6553-4e90-9013-57f222fad5e6\") " pod="openshift-monitoring/metrics-server-6bcb9bcc4b-5wrlm" Jul 08 18:47:17.517264 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.517099 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9b81ec53-6553-4e90-9013-57f222fad5e6-secret-metrics-server-client-certs\") pod \"metrics-server-6bcb9bcc4b-5wrlm\" (UID: \"9b81ec53-6553-4e90-9013-57f222fad5e6\") " pod="openshift-monitoring/metrics-server-6bcb9bcc4b-5wrlm" Jul 08 18:47:17.517724 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.517672 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b81ec53-6553-4e90-9013-57f222fad5e6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6bcb9bcc4b-5wrlm\" (UID: \"9b81ec53-6553-4e90-9013-57f222fad5e6\") " pod="openshift-monitoring/metrics-server-6bcb9bcc4b-5wrlm" Jul 08 18:47:17.518177 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.518129 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9b81ec53-6553-4e90-9013-57f222fad5e6-audit-log\") pod \"metrics-server-6bcb9bcc4b-5wrlm\" (UID: \"9b81ec53-6553-4e90-9013-57f222fad5e6\") " pod="openshift-monitoring/metrics-server-6bcb9bcc4b-5wrlm" Jul 08 18:47:17.518969 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.518921 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9b81ec53-6553-4e90-9013-57f222fad5e6-metrics-server-audit-profiles\") pod \"metrics-server-6bcb9bcc4b-5wrlm\" (UID: \"9b81ec53-6553-4e90-9013-57f222fad5e6\") " pod="openshift-monitoring/metrics-server-6bcb9bcc4b-5wrlm" Jul 08 18:47:17.520868 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.520843 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9b81ec53-6553-4e90-9013-57f222fad5e6-secret-metrics-server-tls\") pod \"metrics-server-6bcb9bcc4b-5wrlm\" (UID: \"9b81ec53-6553-4e90-9013-57f222fad5e6\") " pod="openshift-monitoring/metrics-server-6bcb9bcc4b-5wrlm" Jul 08 18:47:17.521133 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.521090 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9b81ec53-6553-4e90-9013-57f222fad5e6-client-ca-bundle\") pod \"metrics-server-6bcb9bcc4b-5wrlm\" (UID: \"9b81ec53-6553-4e90-9013-57f222fad5e6\") " pod="openshift-monitoring/metrics-server-6bcb9bcc4b-5wrlm" Jul 08 18:47:17.522051 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.522029 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9b81ec53-6553-4e90-9013-57f222fad5e6-secret-metrics-server-client-certs\") pod \"metrics-server-6bcb9bcc4b-5wrlm\" (UID: \"9b81ec53-6553-4e90-9013-57f222fad5e6\") " pod="openshift-monitoring/metrics-server-6bcb9bcc4b-5wrlm" Jul 08 18:47:17.526519 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.526498 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4flcz\" (UniqueName: \"kubernetes.io/projected/9b81ec53-6553-4e90-9013-57f222fad5e6-kube-api-access-4flcz\") pod \"metrics-server-6bcb9bcc4b-5wrlm\" (UID: \"9b81ec53-6553-4e90-9013-57f222fad5e6\") " pod="openshift-monitoring/metrics-server-6bcb9bcc4b-5wrlm" Jul 08 18:47:17.601413 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.601379 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6bcb9bcc4b-5wrlm" Jul 08 18:47:17.664937 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.663668 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-lk2jp"] Jul 08 18:47:17.686340 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.686246 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-69846779c7-s85bh"] Jul 08 18:47:17.686484 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.686382 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-lk2jp" Jul 08 18:47:17.693885 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.693826 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 08 18:47:17.694094 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.694071 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-nhxmn\"" Jul 08 18:47:17.710689 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.710655 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-lk2jp"] Jul 08 18:47:17.710858 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.710841 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69846779c7-s85bh" Jul 08 18:47:17.720641 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.719838 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e05edf7-1d8f-4a6d-b935-4d2d91447b48" path="/var/lib/kubelet/pods/2e05edf7-1d8f-4a6d-b935-4d2d91447b48/volumes" Jul 08 18:47:17.720641 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.720224 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69846779c7-s85bh"] Jul 08 18:47:17.720641 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.720243 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 08 18:47:17.820897 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.820848 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04e41c9a-3b76-40ac-a1de-ac2be44a6570-oauth-serving-cert\") pod \"console-69846779c7-s85bh\" (UID: \"04e41c9a-3b76-40ac-a1de-ac2be44a6570\") " pod="openshift-console/console-69846779c7-s85bh" Jul 08 18:47:17.820897 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.820900 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fhk4\" (UniqueName: \"kubernetes.io/projected/04e41c9a-3b76-40ac-a1de-ac2be44a6570-kube-api-access-5fhk4\") pod \"console-69846779c7-s85bh\" (UID: \"04e41c9a-3b76-40ac-a1de-ac2be44a6570\") " pod="openshift-console/console-69846779c7-s85bh" Jul 08 18:47:17.821152 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.820991 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04e41c9a-3b76-40ac-a1de-ac2be44a6570-console-serving-cert\") pod \"console-69846779c7-s85bh\" (UID: \"04e41c9a-3b76-40ac-a1de-ac2be44a6570\") " pod="openshift-console/console-69846779c7-s85bh" Jul 08 18:47:17.821152 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.821035 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04e41c9a-3b76-40ac-a1de-ac2be44a6570-trusted-ca-bundle\") pod \"console-69846779c7-s85bh\" (UID: \"04e41c9a-3b76-40ac-a1de-ac2be44a6570\") " pod="openshift-console/console-69846779c7-s85bh" Jul 08 18:47:17.821152 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.821112 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04e41c9a-3b76-40ac-a1de-ac2be44a6570-service-ca\") pod \"console-69846779c7-s85bh\" (UID: \"04e41c9a-3b76-40ac-a1de-ac2be44a6570\") " pod="openshift-console/console-69846779c7-s85bh" Jul 08 18:47:17.821152 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.821147 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvg2t\" (UniqueName: \"kubernetes.io/projected/e0b7951b-9599-4e80-bf70-173c5a4cf310-kube-api-access-vvg2t\") pod \"monitoring-plugin-d5dcff7cf-lk2jp\" (UID: \"e0b7951b-9599-4e80-bf70-173c5a4cf310\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-lk2jp" Jul 08 18:47:17.821358 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.821179 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04e41c9a-3b76-40ac-a1de-ac2be44a6570-console-oauth-config\") pod \"console-69846779c7-s85bh\" (UID: \"04e41c9a-3b76-40ac-a1de-ac2be44a6570\") " pod="openshift-console/console-69846779c7-s85bh" Jul 08 18:47:17.821358 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.821216 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e0b7951b-9599-4e80-bf70-173c5a4cf310-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-lk2jp\" (UID: \"e0b7951b-9599-4e80-bf70-173c5a4cf310\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-lk2jp" Jul 08 18:47:17.821358 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.821292 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04e41c9a-3b76-40ac-a1de-ac2be44a6570-console-config\") pod \"console-69846779c7-s85bh\" (UID: \"04e41c9a-3b76-40ac-a1de-ac2be44a6570\") " pod="openshift-console/console-69846779c7-s85bh" Jul 08 18:47:17.922806 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.922761 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e0b7951b-9599-4e80-bf70-173c5a4cf310-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-lk2jp\" (UID: \"e0b7951b-9599-4e80-bf70-173c5a4cf310\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-lk2jp" Jul 08 18:47:17.922986 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.922841 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04e41c9a-3b76-40ac-a1de-ac2be44a6570-console-config\") pod \"console-69846779c7-s85bh\" (UID: \"04e41c9a-3b76-40ac-a1de-ac2be44a6570\") " pod="openshift-console/console-69846779c7-s85bh" Jul 08 18:47:17.922986 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.922921 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04e41c9a-3b76-40ac-a1de-ac2be44a6570-oauth-serving-cert\") pod \"console-69846779c7-s85bh\" (UID: \"04e41c9a-3b76-40ac-a1de-ac2be44a6570\") " pod="openshift-console/console-69846779c7-s85bh" Jul 08 18:47:17.923378 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:47:17.923143 2570 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 08 18:47:17.923378 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:47:17.923224 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0b7951b-9599-4e80-bf70-173c5a4cf310-monitoring-plugin-cert podName:e0b7951b-9599-4e80-bf70-173c5a4cf310 nodeName:}" failed. No retries permitted until 2026-07-08 18:47:18.42320167 +0000 UTC m=+99.270476331 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/e0b7951b-9599-4e80-bf70-173c5a4cf310-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-lk2jp" (UID: "e0b7951b-9599-4e80-bf70-173c5a4cf310") : secret "monitoring-plugin-cert" not found Jul 08 18:47:17.924446 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.923786 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04e41c9a-3b76-40ac-a1de-ac2be44a6570-console-config\") pod \"console-69846779c7-s85bh\" (UID: \"04e41c9a-3b76-40ac-a1de-ac2be44a6570\") " pod="openshift-console/console-69846779c7-s85bh" Jul 08 18:47:17.924446 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.924269 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04e41c9a-3b76-40ac-a1de-ac2be44a6570-oauth-serving-cert\") pod \"console-69846779c7-s85bh\" (UID: \"04e41c9a-3b76-40ac-a1de-ac2be44a6570\") " pod="openshift-console/console-69846779c7-s85bh" Jul 08 18:47:17.924780 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.924749 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fhk4\" (UniqueName: \"kubernetes.io/projected/04e41c9a-3b76-40ac-a1de-ac2be44a6570-kube-api-access-5fhk4\") pod \"console-69846779c7-s85bh\" (UID: \"04e41c9a-3b76-40ac-a1de-ac2be44a6570\") " pod="openshift-console/console-69846779c7-s85bh" Jul 08 18:47:17.924882 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.924826 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04e41c9a-3b76-40ac-a1de-ac2be44a6570-console-serving-cert\") pod \"console-69846779c7-s85bh\" (UID: \"04e41c9a-3b76-40ac-a1de-ac2be44a6570\") " pod="openshift-console/console-69846779c7-s85bh" Jul 08 18:47:17.924882 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.924857 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04e41c9a-3b76-40ac-a1de-ac2be44a6570-trusted-ca-bundle\") pod \"console-69846779c7-s85bh\" (UID: \"04e41c9a-3b76-40ac-a1de-ac2be44a6570\") " pod="openshift-console/console-69846779c7-s85bh" Jul 08 18:47:17.924982 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.924902 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04e41c9a-3b76-40ac-a1de-ac2be44a6570-service-ca\") pod \"console-69846779c7-s85bh\" (UID: \"04e41c9a-3b76-40ac-a1de-ac2be44a6570\") " pod="openshift-console/console-69846779c7-s85bh" Jul 08 18:47:17.924982 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.924934 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvg2t\" (UniqueName: \"kubernetes.io/projected/e0b7951b-9599-4e80-bf70-173c5a4cf310-kube-api-access-vvg2t\") pod \"monitoring-plugin-d5dcff7cf-lk2jp\" (UID: \"e0b7951b-9599-4e80-bf70-173c5a4cf310\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-lk2jp" Jul 08 18:47:17.924982 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.924962 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04e41c9a-3b76-40ac-a1de-ac2be44a6570-console-oauth-config\") pod \"console-69846779c7-s85bh\" (UID: \"04e41c9a-3b76-40ac-a1de-ac2be44a6570\") " pod="openshift-console/console-69846779c7-s85bh" Jul 08 18:47:17.926477 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.926451 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04e41c9a-3b76-40ac-a1de-ac2be44a6570-trusted-ca-bundle\") pod \"console-69846779c7-s85bh\" (UID: \"04e41c9a-3b76-40ac-a1de-ac2be44a6570\") " pod="openshift-console/console-69846779c7-s85bh" Jul 08 18:47:17.927039 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.927011 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04e41c9a-3b76-40ac-a1de-ac2be44a6570-service-ca\") pod \"console-69846779c7-s85bh\" (UID: \"04e41c9a-3b76-40ac-a1de-ac2be44a6570\") " pod="openshift-console/console-69846779c7-s85bh" Jul 08 18:47:17.928381 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.928327 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04e41c9a-3b76-40ac-a1de-ac2be44a6570-console-oauth-config\") pod \"console-69846779c7-s85bh\" (UID: \"04e41c9a-3b76-40ac-a1de-ac2be44a6570\") " pod="openshift-console/console-69846779c7-s85bh" Jul 08 18:47:17.928572 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.928534 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04e41c9a-3b76-40ac-a1de-ac2be44a6570-console-serving-cert\") pod \"console-69846779c7-s85bh\" (UID: \"04e41c9a-3b76-40ac-a1de-ac2be44a6570\") " pod="openshift-console/console-69846779c7-s85bh" Jul 08 18:47:17.934856 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.933932 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fhk4\" (UniqueName: \"kubernetes.io/projected/04e41c9a-3b76-40ac-a1de-ac2be44a6570-kube-api-access-5fhk4\") pod \"console-69846779c7-s85bh\" (UID: \"04e41c9a-3b76-40ac-a1de-ac2be44a6570\") " pod="openshift-console/console-69846779c7-s85bh" Jul 08 18:47:17.942540 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:17.942455 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvg2t\" (UniqueName: \"kubernetes.io/projected/e0b7951b-9599-4e80-bf70-173c5a4cf310-kube-api-access-vvg2t\") pod \"monitoring-plugin-d5dcff7cf-lk2jp\" (UID: \"e0b7951b-9599-4e80-bf70-173c5a4cf310\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-lk2jp" Jul 08 18:47:18.024537 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:18.024466 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69846779c7-s85bh" Jul 08 18:47:18.270908 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:18.270840 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69846779c7-s85bh"] Jul 08 18:47:18.273944 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:47:18.273910 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e41c9a_3b76_40ac_a1de_ac2be44a6570.slice/crio-7f95d01126f2249781c1087dcb332567c26454d0ac1cf3aaa1e02ce5adba5269 WatchSource:0}: Error finding container 7f95d01126f2249781c1087dcb332567c26454d0ac1cf3aaa1e02ce5adba5269: Status 404 returned error can't find the container with id 7f95d01126f2249781c1087dcb332567c26454d0ac1cf3aaa1e02ce5adba5269 Jul 08 18:47:18.294900 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:18.294866 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6bcb9bcc4b-5wrlm"] Jul 08 18:47:18.298885 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:47:18.298856 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b81ec53_6553_4e90_9013_57f222fad5e6.slice/crio-2faa668da3ef9471db56830d0df9a5bbfc8df6c0654d8e278ffe8f216641a934 WatchSource:0}: Error finding container 2faa668da3ef9471db56830d0df9a5bbfc8df6c0654d8e278ffe8f216641a934: Status 404 returned error can't find the container with id 2faa668da3ef9471db56830d0df9a5bbfc8df6c0654d8e278ffe8f216641a934 Jul 08 18:47:18.430878 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:18.430840 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e0b7951b-9599-4e80-bf70-173c5a4cf310-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-lk2jp\" (UID: \"e0b7951b-9599-4e80-bf70-173c5a4cf310\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-lk2jp" Jul 08 18:47:18.433649 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:18.433624 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e0b7951b-9599-4e80-bf70-173c5a4cf310-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-lk2jp\" (UID: \"e0b7951b-9599-4e80-bf70-173c5a4cf310\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-lk2jp" Jul 08 18:47:18.611088 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:18.611062 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-lk2jp" Jul 08 18:47:18.788693 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:18.788403 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-lk2jp"] Jul 08 18:47:18.791310 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:47:18.791248 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b7951b_9599_4e80_bf70_173c5a4cf310.slice/crio-9d31f400c8c9279de6e67063032126903db7997eeed7737b4ecd698050d554ab WatchSource:0}: Error finding container 9d31f400c8c9279de6e67063032126903db7997eeed7737b4ecd698050d554ab: Status 404 returned error can't find the container with id 9d31f400c8c9279de6e67063032126903db7997eeed7737b4ecd698050d554ab Jul 08 18:47:19.162865 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:19.162606 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7756484b49-bg58n" event={"ID":"a41d4b66-d1a1-4393-83a8-de9bbe36c08a","Type":"ContainerStarted","Data":"3384f137aee026a4926134a0eb212ba39454729bbf48dea983283c7fb69818fc"} Jul 08 18:47:19.162865 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:19.162703 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7756484b49-bg58n" event={"ID":"a41d4b66-d1a1-4393-83a8-de9bbe36c08a","Type":"ContainerStarted","Data":"9482b9b2a9ff9dd05deb75d7a55e7fc90f8ef216edb2b67689e7ef0f40c3c750"} Jul 08 18:47:19.162865 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:19.162748 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7756484b49-bg58n" event={"ID":"a41d4b66-d1a1-4393-83a8-de9bbe36c08a","Type":"ContainerStarted","Data":"12ebe02a53a8f6be5ff52b8c26f96d0759fc0cb2be72fe5daa899a0eae3f0f74"} Jul 08 18:47:19.166409 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:19.165545 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69846779c7-s85bh" event={"ID":"04e41c9a-3b76-40ac-a1de-ac2be44a6570","Type":"ContainerStarted","Data":"f0efb09a9646537bbd430d7da33409e5c8149979891bd393d0ca367e49e082c6"} Jul 08 18:47:19.166409 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:19.165605 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69846779c7-s85bh" event={"ID":"04e41c9a-3b76-40ac-a1de-ac2be44a6570","Type":"ContainerStarted","Data":"7f95d01126f2249781c1087dcb332567c26454d0ac1cf3aaa1e02ce5adba5269"} Jul 08 18:47:19.172855 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:19.172819 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-lk2jp" event={"ID":"e0b7951b-9599-4e80-bf70-173c5a4cf310","Type":"ContainerStarted","Data":"9d31f400c8c9279de6e67063032126903db7997eeed7737b4ecd698050d554ab"} Jul 08 18:47:19.175645 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:19.175586 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6bcb9bcc4b-5wrlm" event={"ID":"9b81ec53-6553-4e90-9013-57f222fad5e6","Type":"ContainerStarted","Data":"2faa668da3ef9471db56830d0df9a5bbfc8df6c0654d8e278ffe8f216641a934"} Jul 08 18:47:19.221212 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:19.221144 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-69846779c7-s85bh" podStartSLOduration=2.221123526 podStartE2EDuration="2.221123526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:47:19.219030934 +0000 UTC m=+100.066305602" watchObservedRunningTime="2026-07-08 18:47:19.221123526 +0000 UTC m=+100.068398194" Jul 08 18:47:22.106358 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.106273 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69846779c7-s85bh"] Jul 08 18:47:22.153626 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.153587 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78fb588474-fcb6r"] Jul 08 18:47:22.174176 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.174134 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78fb588474-fcb6r" Jul 08 18:47:22.174551 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.174524 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78fb588474-fcb6r"] Jul 08 18:47:22.191997 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.191965 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7756484b49-bg58n" event={"ID":"a41d4b66-d1a1-4393-83a8-de9bbe36c08a","Type":"ContainerStarted","Data":"242125af941c055a7818109f27e201adc81403c77a48f8143a3c31c5e16fd5c1"} Jul 08 18:47:22.192138 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.192048 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7756484b49-bg58n" event={"ID":"a41d4b66-d1a1-4393-83a8-de9bbe36c08a","Type":"ContainerStarted","Data":"ad57e7eb67ee9e11acf426050f585caeb3fc5ac9fbc317a109197c68cec9609f"} Jul 08 18:47:22.192138 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.192093 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7756484b49-bg58n" event={"ID":"a41d4b66-d1a1-4393-83a8-de9bbe36c08a","Type":"ContainerStarted","Data":"700ac225a2930d4cc4507be958c2a79a38c99a27cc3228544f81c88a6d61f57a"} Jul 08 18:47:22.193645 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.193616 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-lk2jp" event={"ID":"e0b7951b-9599-4e80-bf70-173c5a4cf310","Type":"ContainerStarted","Data":"5c0dc3f682648fb3f2b6a1edc9f929c818dcabb57b7678a81f76f9b22d7f1d29"} Jul 08 18:47:22.193757 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.193668 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-lk2jp" Jul 08 18:47:22.195088 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.195068 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6bcb9bcc4b-5wrlm" event={"ID":"9b81ec53-6553-4e90-9013-57f222fad5e6","Type":"ContainerStarted","Data":"42cdcda66eb62d9395c8ffedd3b08ad14c0524c501861623a961745aedc7329d"} Jul 08 18:47:22.198779 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.198757 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-lk2jp" Jul 08 18:47:22.222932 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.222880 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6bcb9bcc4b-5wrlm" podStartSLOduration=1.9836965549999999 podStartE2EDuration="5.222852676s" totalImagesPullingTime="3.239156121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:47:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:47:22.219552096 +0000 UTC m=+103.066826775" watchObservedRunningTime="2026-07-08 18:47:22.222852676 +0000 UTC m=+103.070127341" Jul 08 18:47:22.248995 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.248954 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7756484b49-bg58n" podStartSLOduration=2.621034772 podStartE2EDuration="8.248940689s" totalImagesPullingTime="5.627905917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:47:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:47:22.2483504 +0000 UTC m=+103.095625090" watchObservedRunningTime="2026-07-08 18:47:22.248940689 +0000 UTC m=+103.096215354" Jul 08 18:47:22.269867 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.269817 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-lk2jp" podStartSLOduration=2.5221762979999998 podStartE2EDuration="5.269802295s" totalImagesPullingTime="2.747625997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:47:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:47:22.268276471 +0000 UTC m=+103.115551129" watchObservedRunningTime="2026-07-08 18:47:22.269802295 +0000 UTC m=+103.117076961" Jul 08 18:47:22.270541 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.270497 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/247914c5-acde-40a2-9b61-464e6eb79ac4-service-ca\") pod \"console-78fb588474-fcb6r\" (UID: \"247914c5-acde-40a2-9b61-464e6eb79ac4\") " pod="openshift-console/console-78fb588474-fcb6r" Jul 08 18:47:22.270678 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.270580 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/247914c5-acde-40a2-9b61-464e6eb79ac4-trusted-ca-bundle\") pod \"console-78fb588474-fcb6r\" (UID: \"247914c5-acde-40a2-9b61-464e6eb79ac4\") " pod="openshift-console/console-78fb588474-fcb6r" Jul 08 18:47:22.270678 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.270664 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/247914c5-acde-40a2-9b61-464e6eb79ac4-oauth-serving-cert\") pod \"console-78fb588474-fcb6r\" (UID: \"247914c5-acde-40a2-9b61-464e6eb79ac4\") " pod="openshift-console/console-78fb588474-fcb6r" Jul 08 18:47:22.270854 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.270822 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmwzm\" (UniqueName: \"kubernetes.io/projected/247914c5-acde-40a2-9b61-464e6eb79ac4-kube-api-access-kmwzm\") pod \"console-78fb588474-fcb6r\" (UID: \"247914c5-acde-40a2-9b61-464e6eb79ac4\") " pod="openshift-console/console-78fb588474-fcb6r" Jul 08 18:47:22.270984 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.270941 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/247914c5-acde-40a2-9b61-464e6eb79ac4-console-serving-cert\") pod \"console-78fb588474-fcb6r\" (UID: \"247914c5-acde-40a2-9b61-464e6eb79ac4\") " pod="openshift-console/console-78fb588474-fcb6r" Jul 08 18:47:22.271044 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.271030 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/247914c5-acde-40a2-9b61-464e6eb79ac4-console-oauth-config\") pod \"console-78fb588474-fcb6r\" (UID: \"247914c5-acde-40a2-9b61-464e6eb79ac4\") " pod="openshift-console/console-78fb588474-fcb6r" Jul 08 18:47:22.271146 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.271109 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/247914c5-acde-40a2-9b61-464e6eb79ac4-console-config\") pod \"console-78fb588474-fcb6r\" (UID: \"247914c5-acde-40a2-9b61-464e6eb79ac4\") " pod="openshift-console/console-78fb588474-fcb6r" Jul 08 18:47:22.372144 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.372110 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/247914c5-acde-40a2-9b61-464e6eb79ac4-service-ca\") pod \"console-78fb588474-fcb6r\" (UID: \"247914c5-acde-40a2-9b61-464e6eb79ac4\") " pod="openshift-console/console-78fb588474-fcb6r" Jul 08 18:47:22.372498 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.372478 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/247914c5-acde-40a2-9b61-464e6eb79ac4-trusted-ca-bundle\") pod \"console-78fb588474-fcb6r\" (UID: \"247914c5-acde-40a2-9b61-464e6eb79ac4\") " pod="openshift-console/console-78fb588474-fcb6r" Jul 08 18:47:22.373436 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.373416 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/247914c5-acde-40a2-9b61-464e6eb79ac4-oauth-serving-cert\") pod \"console-78fb588474-fcb6r\" (UID: \"247914c5-acde-40a2-9b61-464e6eb79ac4\") " pod="openshift-console/console-78fb588474-fcb6r" Jul 08 18:47:22.373588 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.373575 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmwzm\" (UniqueName: \"kubernetes.io/projected/247914c5-acde-40a2-9b61-464e6eb79ac4-kube-api-access-kmwzm\") pod \"console-78fb588474-fcb6r\" (UID: \"247914c5-acde-40a2-9b61-464e6eb79ac4\") " pod="openshift-console/console-78fb588474-fcb6r" Jul 08 18:47:22.373712 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.373700 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/247914c5-acde-40a2-9b61-464e6eb79ac4-console-serving-cert\") pod \"console-78fb588474-fcb6r\" (UID: \"247914c5-acde-40a2-9b61-464e6eb79ac4\") " pod="openshift-console/console-78fb588474-fcb6r" Jul 08 18:47:22.373835 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.373812 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/247914c5-acde-40a2-9b61-464e6eb79ac4-console-oauth-config\") pod \"console-78fb588474-fcb6r\" (UID: \"247914c5-acde-40a2-9b61-464e6eb79ac4\") " pod="openshift-console/console-78fb588474-fcb6r" Jul 08 18:47:22.373958 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.373845 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/247914c5-acde-40a2-9b61-464e6eb79ac4-console-config\") pod \"console-78fb588474-fcb6r\" (UID: \"247914c5-acde-40a2-9b61-464e6eb79ac4\") " pod="openshift-console/console-78fb588474-fcb6r" Jul 08 18:47:22.374111 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.373356 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/247914c5-acde-40a2-9b61-464e6eb79ac4-trusted-ca-bundle\") pod \"console-78fb588474-fcb6r\" (UID: \"247914c5-acde-40a2-9b61-464e6eb79ac4\") " pod="openshift-console/console-78fb588474-fcb6r" Jul 08 18:47:22.374215 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.373105 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/247914c5-acde-40a2-9b61-464e6eb79ac4-service-ca\") pod \"console-78fb588474-fcb6r\" (UID: \"247914c5-acde-40a2-9b61-464e6eb79ac4\") " pod="openshift-console/console-78fb588474-fcb6r" Jul 08 18:47:22.374555 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.374519 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/247914c5-acde-40a2-9b61-464e6eb79ac4-console-config\") pod \"console-78fb588474-fcb6r\" (UID: \"247914c5-acde-40a2-9b61-464e6eb79ac4\") " pod="openshift-console/console-78fb588474-fcb6r" Jul 08 18:47:22.374827 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.374778 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/247914c5-acde-40a2-9b61-464e6eb79ac4-oauth-serving-cert\") pod \"console-78fb588474-fcb6r\" (UID: \"247914c5-acde-40a2-9b61-464e6eb79ac4\") " pod="openshift-console/console-78fb588474-fcb6r" Jul 08 18:47:22.376535 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.376515 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/247914c5-acde-40a2-9b61-464e6eb79ac4-console-serving-cert\") pod \"console-78fb588474-fcb6r\" (UID: \"247914c5-acde-40a2-9b61-464e6eb79ac4\") " pod="openshift-console/console-78fb588474-fcb6r" Jul 08 18:47:22.376853 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.376835 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/247914c5-acde-40a2-9b61-464e6eb79ac4-console-oauth-config\") pod \"console-78fb588474-fcb6r\" (UID: \"247914c5-acde-40a2-9b61-464e6eb79ac4\") " pod="openshift-console/console-78fb588474-fcb6r" Jul 08 18:47:22.382643 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.382625 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmwzm\" (UniqueName: \"kubernetes.io/projected/247914c5-acde-40a2-9b61-464e6eb79ac4-kube-api-access-kmwzm\") pod \"console-78fb588474-fcb6r\" (UID: \"247914c5-acde-40a2-9b61-464e6eb79ac4\") " pod="openshift-console/console-78fb588474-fcb6r" Jul 08 18:47:22.484978 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.484949 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78fb588474-fcb6r" Jul 08 18:47:22.647738 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:22.647664 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78fb588474-fcb6r"] Jul 08 18:47:22.650455 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:47:22.650423 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod247914c5_acde_40a2_9b61_464e6eb79ac4.slice/crio-b0083f2b3070f2992c05acf1d74642c1980ebe6f34824a714e435ef1fdd48020 WatchSource:0}: Error finding container b0083f2b3070f2992c05acf1d74642c1980ebe6f34824a714e435ef1fdd48020: Status 404 returned error can't find the container with id b0083f2b3070f2992c05acf1d74642c1980ebe6f34824a714e435ef1fdd48020 Jul 08 18:47:23.005730 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:23.005655 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mmgzx" Jul 08 18:47:23.200769 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:23.200733 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78fb588474-fcb6r" event={"ID":"247914c5-acde-40a2-9b61-464e6eb79ac4","Type":"ContainerStarted","Data":"c109dd2a4d3674cfce978ea141a04c06092e7a0f372f7c2fd19e59aa876cca8e"} Jul 08 18:47:23.201200 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:23.200792 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78fb588474-fcb6r" event={"ID":"247914c5-acde-40a2-9b61-464e6eb79ac4","Type":"ContainerStarted","Data":"b0083f2b3070f2992c05acf1d74642c1980ebe6f34824a714e435ef1fdd48020"} Jul 08 18:47:23.201532 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:23.201514 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7756484b49-bg58n" Jul 08 18:47:23.207445 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:23.207422 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7756484b49-bg58n" Jul 08 18:47:23.221145 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:23.221093 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78fb588474-fcb6r" podStartSLOduration=1.221075332 podStartE2EDuration="1.221075332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:47:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:47:23.219407089 +0000 UTC m=+104.066681754" watchObservedRunningTime="2026-07-08 18:47:23.221075332 +0000 UTC m=+104.068349998" Jul 08 18:47:28.025269 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:28.025226 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69846779c7-s85bh" Jul 08 18:47:32.485811 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:32.485773 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78fb588474-fcb6r" Jul 08 18:47:32.486180 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:32.485827 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78fb588474-fcb6r" Jul 08 18:47:32.491414 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:32.491394 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78fb588474-fcb6r" Jul 08 18:47:33.243513 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:33.243489 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78fb588474-fcb6r" Jul 08 18:47:37.123502 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:37.123439 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5db8cf5545-hzsdv" podUID="1b8f10ac-84f9-4e89-91ac-08dbc619b666" containerName="console" containerID="cri-o://83cdc8bdf201e680a60ec4b41e3c3c649fe9907818e4724c05633bc580c19f35" gracePeriod=15 Jul 08 18:47:37.256603 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:37.256574 2570 generic.go:356] "Generic (PLEG): container finished" podID="1b8f10ac-84f9-4e89-91ac-08dbc619b666" containerID="83cdc8bdf201e680a60ec4b41e3c3c649fe9907818e4724c05633bc580c19f35" exitCode=0 Jul 08 18:47:37.256727 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:37.256641 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5db8cf5545-hzsdv" event={"ID":"1b8f10ac-84f9-4e89-91ac-08dbc619b666","Type":"ContainerDied","Data":"83cdc8bdf201e680a60ec4b41e3c3c649fe9907818e4724c05633bc580c19f35"} Jul 08 18:47:37.400835 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:37.400772 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5db8cf5545-hzsdv" Jul 08 18:47:37.501279 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:37.501220 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b8f10ac-84f9-4e89-91ac-08dbc619b666-oauth-serving-cert\") pod \"1b8f10ac-84f9-4e89-91ac-08dbc619b666\" (UID: \"1b8f10ac-84f9-4e89-91ac-08dbc619b666\") " Jul 08 18:47:37.501449 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:37.501291 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b8f10ac-84f9-4e89-91ac-08dbc619b666-console-oauth-config\") pod \"1b8f10ac-84f9-4e89-91ac-08dbc619b666\" (UID: \"1b8f10ac-84f9-4e89-91ac-08dbc619b666\") " Jul 08 18:47:37.501449 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:37.501324 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b8f10ac-84f9-4e89-91ac-08dbc619b666-service-ca\") pod \"1b8f10ac-84f9-4e89-91ac-08dbc619b666\" (UID: \"1b8f10ac-84f9-4e89-91ac-08dbc619b666\") " Jul 08 18:47:37.501449 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:37.501392 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b8f10ac-84f9-4e89-91ac-08dbc619b666-console-serving-cert\") pod \"1b8f10ac-84f9-4e89-91ac-08dbc619b666\" (UID: \"1b8f10ac-84f9-4e89-91ac-08dbc619b666\") " Jul 08 18:47:37.501449 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:37.501424 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b8f10ac-84f9-4e89-91ac-08dbc619b666-console-config\") pod \"1b8f10ac-84f9-4e89-91ac-08dbc619b666\" (UID: \"1b8f10ac-84f9-4e89-91ac-08dbc619b666\") " Jul 08 18:47:37.501665 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:37.501451 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzbh4\" (UniqueName: \"kubernetes.io/projected/1b8f10ac-84f9-4e89-91ac-08dbc619b666-kube-api-access-xzbh4\") pod \"1b8f10ac-84f9-4e89-91ac-08dbc619b666\" (UID: \"1b8f10ac-84f9-4e89-91ac-08dbc619b666\") " Jul 08 18:47:37.501731 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:37.501706 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b8f10ac-84f9-4e89-91ac-08dbc619b666-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1b8f10ac-84f9-4e89-91ac-08dbc619b666" (UID: "1b8f10ac-84f9-4e89-91ac-08dbc619b666"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:47:37.501821 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:37.501798 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b8f10ac-84f9-4e89-91ac-08dbc619b666-service-ca" (OuterVolumeSpecName: "service-ca") pod "1b8f10ac-84f9-4e89-91ac-08dbc619b666" (UID: "1b8f10ac-84f9-4e89-91ac-08dbc619b666"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:47:37.501883 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:37.501864 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b8f10ac-84f9-4e89-91ac-08dbc619b666-console-config" (OuterVolumeSpecName: "console-config") pod "1b8f10ac-84f9-4e89-91ac-08dbc619b666" (UID: "1b8f10ac-84f9-4e89-91ac-08dbc619b666"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:47:37.503759 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:37.503730 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b8f10ac-84f9-4e89-91ac-08dbc619b666-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1b8f10ac-84f9-4e89-91ac-08dbc619b666" (UID: "1b8f10ac-84f9-4e89-91ac-08dbc619b666"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:47:37.503868 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:37.503749 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b8f10ac-84f9-4e89-91ac-08dbc619b666-kube-api-access-xzbh4" (OuterVolumeSpecName: "kube-api-access-xzbh4") pod "1b8f10ac-84f9-4e89-91ac-08dbc619b666" (UID: "1b8f10ac-84f9-4e89-91ac-08dbc619b666"). InnerVolumeSpecName "kube-api-access-xzbh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:47:37.503868 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:37.503790 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b8f10ac-84f9-4e89-91ac-08dbc619b666-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1b8f10ac-84f9-4e89-91ac-08dbc619b666" (UID: "1b8f10ac-84f9-4e89-91ac-08dbc619b666"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:47:37.601922 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:37.601894 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6bcb9bcc4b-5wrlm" Jul 08 18:47:37.602126 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:37.601954 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6bcb9bcc4b-5wrlm" Jul 08 18:47:37.602294 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:37.602279 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b8f10ac-84f9-4e89-91ac-08dbc619b666-console-serving-cert\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:47:37.602340 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:37.602312 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b8f10ac-84f9-4e89-91ac-08dbc619b666-console-config\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:47:37.602340 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:37.602338 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzbh4\" (UniqueName: \"kubernetes.io/projected/1b8f10ac-84f9-4e89-91ac-08dbc619b666-kube-api-access-xzbh4\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:47:37.602409 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:37.602353 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b8f10ac-84f9-4e89-91ac-08dbc619b666-oauth-serving-cert\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:47:37.602409 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:37.602367 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b8f10ac-84f9-4e89-91ac-08dbc619b666-console-oauth-config\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:47:37.602409 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:37.602384 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b8f10ac-84f9-4e89-91ac-08dbc619b666-service-ca\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:47:38.261490 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:38.261452 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5db8cf5545-hzsdv" event={"ID":"1b8f10ac-84f9-4e89-91ac-08dbc619b666","Type":"ContainerDied","Data":"142d78bb8fff98a02d73f1fa71d6316d0d554d77281a3995c482ceb05ddf45b8"} Jul 08 18:47:38.262001 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:38.261535 2570 scope.go:117] "RemoveContainer" containerID="83cdc8bdf201e680a60ec4b41e3c3c649fe9907818e4724c05633bc580c19f35" Jul 08 18:47:38.262001 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:38.261466 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5db8cf5545-hzsdv" Jul 08 18:47:38.278330 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:38.278294 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5db8cf5545-hzsdv"] Jul 08 18:47:38.281943 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:38.281922 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5db8cf5545-hzsdv"] Jul 08 18:47:39.688914 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:39.688883 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b8f10ac-84f9-4e89-91ac-08dbc619b666" path="/var/lib/kubelet/pods/1b8f10ac-84f9-4e89-91ac-08dbc619b666/volumes" Jul 08 18:47:47.129207 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:47.129163 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-69846779c7-s85bh" podUID="04e41c9a-3b76-40ac-a1de-ac2be44a6570" containerName="console" containerID="cri-o://f0efb09a9646537bbd430d7da33409e5c8149979891bd393d0ca367e49e082c6" gracePeriod=15 Jul 08 18:47:47.301448 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:47.301416 2570 generic.go:356] "Generic (PLEG): container finished" podID="04e41c9a-3b76-40ac-a1de-ac2be44a6570" containerID="f0efb09a9646537bbd430d7da33409e5c8149979891bd393d0ca367e49e082c6" exitCode=0 Jul 08 18:47:47.301616 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:47.301465 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69846779c7-s85bh" event={"ID":"04e41c9a-3b76-40ac-a1de-ac2be44a6570","Type":"ContainerDied","Data":"f0efb09a9646537bbd430d7da33409e5c8149979891bd393d0ca367e49e082c6"} Jul 08 18:47:47.461193 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:47.461166 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69846779c7-s85bh" Jul 08 18:47:47.590711 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:47.590673 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04e41c9a-3b76-40ac-a1de-ac2be44a6570-console-config\") pod \"04e41c9a-3b76-40ac-a1de-ac2be44a6570\" (UID: \"04e41c9a-3b76-40ac-a1de-ac2be44a6570\") " Jul 08 18:47:47.590881 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:47.590761 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04e41c9a-3b76-40ac-a1de-ac2be44a6570-console-oauth-config\") pod \"04e41c9a-3b76-40ac-a1de-ac2be44a6570\" (UID: \"04e41c9a-3b76-40ac-a1de-ac2be44a6570\") " Jul 08 18:47:47.590881 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:47.590788 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04e41c9a-3b76-40ac-a1de-ac2be44a6570-service-ca\") pod \"04e41c9a-3b76-40ac-a1de-ac2be44a6570\" (UID: \"04e41c9a-3b76-40ac-a1de-ac2be44a6570\") " Jul 08 18:47:47.590881 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:47.590847 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04e41c9a-3b76-40ac-a1de-ac2be44a6570-trusted-ca-bundle\") pod \"04e41c9a-3b76-40ac-a1de-ac2be44a6570\" (UID: \"04e41c9a-3b76-40ac-a1de-ac2be44a6570\") " Jul 08 18:47:47.591052 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:47.590884 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04e41c9a-3b76-40ac-a1de-ac2be44a6570-oauth-serving-cert\") pod \"04e41c9a-3b76-40ac-a1de-ac2be44a6570\" (UID: \"04e41c9a-3b76-40ac-a1de-ac2be44a6570\") " Jul 08 18:47:47.591052 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:47.590913 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04e41c9a-3b76-40ac-a1de-ac2be44a6570-console-serving-cert\") pod \"04e41c9a-3b76-40ac-a1de-ac2be44a6570\" (UID: \"04e41c9a-3b76-40ac-a1de-ac2be44a6570\") " Jul 08 18:47:47.591052 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:47.590938 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fhk4\" (UniqueName: \"kubernetes.io/projected/04e41c9a-3b76-40ac-a1de-ac2be44a6570-kube-api-access-5fhk4\") pod \"04e41c9a-3b76-40ac-a1de-ac2be44a6570\" (UID: \"04e41c9a-3b76-40ac-a1de-ac2be44a6570\") " Jul 08 18:47:47.591202 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:47.591053 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04e41c9a-3b76-40ac-a1de-ac2be44a6570-console-config" (OuterVolumeSpecName: "console-config") pod "04e41c9a-3b76-40ac-a1de-ac2be44a6570" (UID: "04e41c9a-3b76-40ac-a1de-ac2be44a6570"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:47:47.591274 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:47.591197 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04e41c9a-3b76-40ac-a1de-ac2be44a6570-console-config\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:47:47.591400 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:47.591376 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04e41c9a-3b76-40ac-a1de-ac2be44a6570-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "04e41c9a-3b76-40ac-a1de-ac2be44a6570" (UID: "04e41c9a-3b76-40ac-a1de-ac2be44a6570"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:47:47.591461 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:47.591439 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04e41c9a-3b76-40ac-a1de-ac2be44a6570-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "04e41c9a-3b76-40ac-a1de-ac2be44a6570" (UID: "04e41c9a-3b76-40ac-a1de-ac2be44a6570"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:47:47.591730 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:47.591706 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04e41c9a-3b76-40ac-a1de-ac2be44a6570-service-ca" (OuterVolumeSpecName: "service-ca") pod "04e41c9a-3b76-40ac-a1de-ac2be44a6570" (UID: "04e41c9a-3b76-40ac-a1de-ac2be44a6570"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:47:47.592984 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:47.592955 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04e41c9a-3b76-40ac-a1de-ac2be44a6570-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "04e41c9a-3b76-40ac-a1de-ac2be44a6570" (UID: "04e41c9a-3b76-40ac-a1de-ac2be44a6570"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:47:47.593058 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:47.593017 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04e41c9a-3b76-40ac-a1de-ac2be44a6570-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "04e41c9a-3b76-40ac-a1de-ac2be44a6570" (UID: "04e41c9a-3b76-40ac-a1de-ac2be44a6570"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:47:47.593140 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:47.593120 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e41c9a-3b76-40ac-a1de-ac2be44a6570-kube-api-access-5fhk4" (OuterVolumeSpecName: "kube-api-access-5fhk4") pod "04e41c9a-3b76-40ac-a1de-ac2be44a6570" (UID: "04e41c9a-3b76-40ac-a1de-ac2be44a6570"). InnerVolumeSpecName "kube-api-access-5fhk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:47:47.692655 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:47.692583 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04e41c9a-3b76-40ac-a1de-ac2be44a6570-console-oauth-config\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:47:47.692655 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:47.692619 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04e41c9a-3b76-40ac-a1de-ac2be44a6570-service-ca\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:47:47.692655 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:47.692636 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04e41c9a-3b76-40ac-a1de-ac2be44a6570-trusted-ca-bundle\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:47:47.692655 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:47.692652 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04e41c9a-3b76-40ac-a1de-ac2be44a6570-oauth-serving-cert\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:47:47.692879 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:47.692666 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04e41c9a-3b76-40ac-a1de-ac2be44a6570-console-serving-cert\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:47:47.692879 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:47.692681 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fhk4\" (UniqueName: \"kubernetes.io/projected/04e41c9a-3b76-40ac-a1de-ac2be44a6570-kube-api-access-5fhk4\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:47:48.307205 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:48.307170 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69846779c7-s85bh" event={"ID":"04e41c9a-3b76-40ac-a1de-ac2be44a6570","Type":"ContainerDied","Data":"7f95d01126f2249781c1087dcb332567c26454d0ac1cf3aaa1e02ce5adba5269"} Jul 08 18:47:48.307205 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:48.307210 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69846779c7-s85bh" Jul 08 18:47:48.307689 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:48.307231 2570 scope.go:117] "RemoveContainer" containerID="f0efb09a9646537bbd430d7da33409e5c8149979891bd393d0ca367e49e082c6" Jul 08 18:47:48.324483 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:48.324461 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69846779c7-s85bh"] Jul 08 18:47:48.328151 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:48.328130 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-69846779c7-s85bh"] Jul 08 18:47:49.688691 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:49.688658 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04e41c9a-3b76-40ac-a1de-ac2be44a6570" path="/var/lib/kubelet/pods/04e41c9a-3b76-40ac-a1de-ac2be44a6570/volumes" Jul 08 18:47:57.607434 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:57.607398 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6bcb9bcc4b-5wrlm" Jul 08 18:47:57.611233 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:47:57.611209 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6bcb9bcc4b-5wrlm" Jul 08 18:48:44.597941 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:48:44.597906 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78fb588474-fcb6r"] Jul 08 18:49:09.617897 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:49:09.617827 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-78fb588474-fcb6r" podUID="247914c5-acde-40a2-9b61-464e6eb79ac4" containerName="console" containerID="cri-o://c109dd2a4d3674cfce978ea141a04c06092e7a0f372f7c2fd19e59aa876cca8e" gracePeriod=15 Jul 08 18:49:09.864486 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:49:09.864464 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78fb588474-fcb6r" Jul 08 18:49:09.946176 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:49:09.946095 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/247914c5-acde-40a2-9b61-464e6eb79ac4-trusted-ca-bundle\") pod \"247914c5-acde-40a2-9b61-464e6eb79ac4\" (UID: \"247914c5-acde-40a2-9b61-464e6eb79ac4\") " Jul 08 18:49:09.946176 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:49:09.946133 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/247914c5-acde-40a2-9b61-464e6eb79ac4-service-ca\") pod \"247914c5-acde-40a2-9b61-464e6eb79ac4\" (UID: \"247914c5-acde-40a2-9b61-464e6eb79ac4\") " Jul 08 18:49:09.946176 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:49:09.946160 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/247914c5-acde-40a2-9b61-464e6eb79ac4-oauth-serving-cert\") pod \"247914c5-acde-40a2-9b61-464e6eb79ac4\" (UID: \"247914c5-acde-40a2-9b61-464e6eb79ac4\") " Jul 08 18:49:09.946444 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:49:09.946188 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/247914c5-acde-40a2-9b61-464e6eb79ac4-console-oauth-config\") pod \"247914c5-acde-40a2-9b61-464e6eb79ac4\" (UID: \"247914c5-acde-40a2-9b61-464e6eb79ac4\") " Jul 08 18:49:09.946444 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:49:09.946216 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmwzm\" (UniqueName: \"kubernetes.io/projected/247914c5-acde-40a2-9b61-464e6eb79ac4-kube-api-access-kmwzm\") pod \"247914c5-acde-40a2-9b61-464e6eb79ac4\" (UID: \"247914c5-acde-40a2-9b61-464e6eb79ac4\") " Jul 08 18:49:09.946444 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:49:09.946282 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/247914c5-acde-40a2-9b61-464e6eb79ac4-console-serving-cert\") pod \"247914c5-acde-40a2-9b61-464e6eb79ac4\" (UID: \"247914c5-acde-40a2-9b61-464e6eb79ac4\") " Jul 08 18:49:09.946613 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:49:09.946582 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/247914c5-acde-40a2-9b61-464e6eb79ac4-service-ca" (OuterVolumeSpecName: "service-ca") pod "247914c5-acde-40a2-9b61-464e6eb79ac4" (UID: "247914c5-acde-40a2-9b61-464e6eb79ac4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:49:09.946653 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:49:09.946594 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/247914c5-acde-40a2-9b61-464e6eb79ac4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "247914c5-acde-40a2-9b61-464e6eb79ac4" (UID: "247914c5-acde-40a2-9b61-464e6eb79ac4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:49:09.946693 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:49:09.946640 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/247914c5-acde-40a2-9b61-464e6eb79ac4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "247914c5-acde-40a2-9b61-464e6eb79ac4" (UID: "247914c5-acde-40a2-9b61-464e6eb79ac4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:49:09.948508 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:49:09.948485 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/247914c5-acde-40a2-9b61-464e6eb79ac4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "247914c5-acde-40a2-9b61-464e6eb79ac4" (UID: "247914c5-acde-40a2-9b61-464e6eb79ac4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:49:09.948601 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:49:09.948528 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/247914c5-acde-40a2-9b61-464e6eb79ac4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "247914c5-acde-40a2-9b61-464e6eb79ac4" (UID: "247914c5-acde-40a2-9b61-464e6eb79ac4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:49:09.948652 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:49:09.948603 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/247914c5-acde-40a2-9b61-464e6eb79ac4-kube-api-access-kmwzm" (OuterVolumeSpecName: "kube-api-access-kmwzm") pod "247914c5-acde-40a2-9b61-464e6eb79ac4" (UID: "247914c5-acde-40a2-9b61-464e6eb79ac4"). InnerVolumeSpecName "kube-api-access-kmwzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:49:10.046773 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:49:10.046728 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/247914c5-acde-40a2-9b61-464e6eb79ac4-console-config\") pod \"247914c5-acde-40a2-9b61-464e6eb79ac4\" (UID: \"247914c5-acde-40a2-9b61-464e6eb79ac4\") " Jul 08 18:49:10.046996 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:49:10.046976 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/247914c5-acde-40a2-9b61-464e6eb79ac4-trusted-ca-bundle\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:49:10.047039 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:49:10.047012 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/247914c5-acde-40a2-9b61-464e6eb79ac4-service-ca\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:49:10.047039 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:49:10.047022 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/247914c5-acde-40a2-9b61-464e6eb79ac4-oauth-serving-cert\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:49:10.047039 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:49:10.047031 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/247914c5-acde-40a2-9b61-464e6eb79ac4-console-oauth-config\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:49:10.047129 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:49:10.047039 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmwzm\" (UniqueName: \"kubernetes.io/projected/247914c5-acde-40a2-9b61-464e6eb79ac4-kube-api-access-kmwzm\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:49:10.047129 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:49:10.047048 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/247914c5-acde-40a2-9b61-464e6eb79ac4-console-serving-cert\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:49:10.047129 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:49:10.047059 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/247914c5-acde-40a2-9b61-464e6eb79ac4-console-config" (OuterVolumeSpecName: "console-config") pod "247914c5-acde-40a2-9b61-464e6eb79ac4" (UID: "247914c5-acde-40a2-9b61-464e6eb79ac4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:49:10.148045 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:49:10.148016 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/247914c5-acde-40a2-9b61-464e6eb79ac4-console-config\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:49:10.614089 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:49:10.614051 2570 generic.go:356] "Generic (PLEG): container finished" podID="247914c5-acde-40a2-9b61-464e6eb79ac4" containerID="c109dd2a4d3674cfce978ea141a04c06092e7a0f372f7c2fd19e59aa876cca8e" exitCode=0 Jul 08 18:49:10.614284 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:49:10.614100 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78fb588474-fcb6r" Jul 08 18:49:10.614284 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:49:10.614134 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78fb588474-fcb6r" event={"ID":"247914c5-acde-40a2-9b61-464e6eb79ac4","Type":"ContainerDied","Data":"c109dd2a4d3674cfce978ea141a04c06092e7a0f372f7c2fd19e59aa876cca8e"} Jul 08 18:49:10.614284 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:49:10.614186 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78fb588474-fcb6r" event={"ID":"247914c5-acde-40a2-9b61-464e6eb79ac4","Type":"ContainerDied","Data":"b0083f2b3070f2992c05acf1d74642c1980ebe6f34824a714e435ef1fdd48020"} Jul 08 18:49:10.614284 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:49:10.614207 2570 scope.go:117] "RemoveContainer" containerID="c109dd2a4d3674cfce978ea141a04c06092e7a0f372f7c2fd19e59aa876cca8e" Jul 08 18:49:10.622810 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:49:10.622619 2570 scope.go:117] "RemoveContainer" containerID="c109dd2a4d3674cfce978ea141a04c06092e7a0f372f7c2fd19e59aa876cca8e" Jul 08 18:49:10.623028 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:49:10.622867 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c109dd2a4d3674cfce978ea141a04c06092e7a0f372f7c2fd19e59aa876cca8e\": container with ID starting with c109dd2a4d3674cfce978ea141a04c06092e7a0f372f7c2fd19e59aa876cca8e not found: ID does not exist" containerID="c109dd2a4d3674cfce978ea141a04c06092e7a0f372f7c2fd19e59aa876cca8e" Jul 08 18:49:10.623028 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:49:10.622893 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c109dd2a4d3674cfce978ea141a04c06092e7a0f372f7c2fd19e59aa876cca8e"} err="failed to get container status \"c109dd2a4d3674cfce978ea141a04c06092e7a0f372f7c2fd19e59aa876cca8e\": rpc error: code = NotFound desc = could not find container \"c109dd2a4d3674cfce978ea141a04c06092e7a0f372f7c2fd19e59aa876cca8e\": container with ID starting with c109dd2a4d3674cfce978ea141a04c06092e7a0f372f7c2fd19e59aa876cca8e not found: ID does not exist" Jul 08 18:49:10.637506 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:49:10.637482 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78fb588474-fcb6r"] Jul 08 18:49:10.644682 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:49:10.644663 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78fb588474-fcb6r"] Jul 08 18:49:11.688427 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:49:11.688396 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="247914c5-acde-40a2-9b61-464e6eb79ac4" path="/var/lib/kubelet/pods/247914c5-acde-40a2-9b61-464e6eb79ac4/volumes" Jul 08 18:50:00.641946 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:00.641874 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754045f7k"] Jul 08 18:50:00.642404 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:00.642143 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04e41c9a-3b76-40ac-a1de-ac2be44a6570" containerName="console" Jul 08 18:50:00.642404 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:00.642153 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="04e41c9a-3b76-40ac-a1de-ac2be44a6570" containerName="console" Jul 08 18:50:00.642404 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:00.642169 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="247914c5-acde-40a2-9b61-464e6eb79ac4" containerName="console" Jul 08 18:50:00.642404 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:00.642174 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="247914c5-acde-40a2-9b61-464e6eb79ac4" containerName="console" Jul 08 18:50:00.642404 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:00.642184 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b8f10ac-84f9-4e89-91ac-08dbc619b666" containerName="console" Jul 08 18:50:00.642404 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:00.642189 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b8f10ac-84f9-4e89-91ac-08dbc619b666" containerName="console" Jul 08 18:50:00.642404 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:00.642239 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="04e41c9a-3b76-40ac-a1de-ac2be44a6570" containerName="console" Jul 08 18:50:00.642404 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:00.642246 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="247914c5-acde-40a2-9b61-464e6eb79ac4" containerName="console" Jul 08 18:50:00.642404 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:00.642268 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b8f10ac-84f9-4e89-91ac-08dbc619b666" containerName="console" Jul 08 18:50:00.644213 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:00.644197 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754045f7k" Jul 08 18:50:00.646604 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:00.646583 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 08 18:50:00.647624 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:00.647609 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rftlb\"" Jul 08 18:50:00.647675 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:00.647639 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 08 18:50:00.654901 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:00.654869 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754045f7k"] Jul 08 18:50:00.739106 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:00.739073 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e111e6ad-cb37-40db-aaa1-a5ae5cea6331-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754045f7k\" (UID: \"e111e6ad-cb37-40db-aaa1-a5ae5cea6331\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754045f7k" Jul 08 18:50:00.739283 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:00.739111 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94ftv\" (UniqueName: \"kubernetes.io/projected/e111e6ad-cb37-40db-aaa1-a5ae5cea6331-kube-api-access-94ftv\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754045f7k\" (UID: \"e111e6ad-cb37-40db-aaa1-a5ae5cea6331\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754045f7k" Jul 08 18:50:00.739283 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:00.739179 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e111e6ad-cb37-40db-aaa1-a5ae5cea6331-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754045f7k\" (UID: \"e111e6ad-cb37-40db-aaa1-a5ae5cea6331\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754045f7k" Jul 08 18:50:00.840192 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:00.840163 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e111e6ad-cb37-40db-aaa1-a5ae5cea6331-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754045f7k\" (UID: \"e111e6ad-cb37-40db-aaa1-a5ae5cea6331\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754045f7k" Jul 08 18:50:00.840350 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:00.840205 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e111e6ad-cb37-40db-aaa1-a5ae5cea6331-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754045f7k\" (UID: \"e111e6ad-cb37-40db-aaa1-a5ae5cea6331\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754045f7k" Jul 08 18:50:00.840350 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:00.840309 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94ftv\" (UniqueName: \"kubernetes.io/projected/e111e6ad-cb37-40db-aaa1-a5ae5cea6331-kube-api-access-94ftv\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754045f7k\" (UID: \"e111e6ad-cb37-40db-aaa1-a5ae5cea6331\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754045f7k" Jul 08 18:50:00.840508 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:00.840489 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e111e6ad-cb37-40db-aaa1-a5ae5cea6331-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754045f7k\" (UID: \"e111e6ad-cb37-40db-aaa1-a5ae5cea6331\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754045f7k" Jul 08 18:50:00.840594 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:00.840574 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e111e6ad-cb37-40db-aaa1-a5ae5cea6331-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754045f7k\" (UID: \"e111e6ad-cb37-40db-aaa1-a5ae5cea6331\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754045f7k" Jul 08 18:50:00.848920 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:00.848897 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94ftv\" (UniqueName: \"kubernetes.io/projected/e111e6ad-cb37-40db-aaa1-a5ae5cea6331-kube-api-access-94ftv\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754045f7k\" (UID: \"e111e6ad-cb37-40db-aaa1-a5ae5cea6331\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754045f7k" Jul 08 18:50:00.953071 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:00.953009 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754045f7k" Jul 08 18:50:01.080086 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:01.080063 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754045f7k"] Jul 08 18:50:01.082559 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:50:01.082527 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode111e6ad_cb37_40db_aaa1_a5ae5cea6331.slice/crio-4863161d1e913cdc5df6deac0c7709957cf7a17c5477877e9f943448c4a7f575 WatchSource:0}: Error finding container 4863161d1e913cdc5df6deac0c7709957cf7a17c5477877e9f943448c4a7f575: Status 404 returned error can't find the container with id 4863161d1e913cdc5df6deac0c7709957cf7a17c5477877e9f943448c4a7f575 Jul 08 18:50:01.796202 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:01.796156 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754045f7k" event={"ID":"e111e6ad-cb37-40db-aaa1-a5ae5cea6331","Type":"ContainerStarted","Data":"4863161d1e913cdc5df6deac0c7709957cf7a17c5477877e9f943448c4a7f575"} Jul 08 18:50:09.830497 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:09.830397 2570 generic.go:356] "Generic (PLEG): container finished" podID="e111e6ad-cb37-40db-aaa1-a5ae5cea6331" containerID="5ec6aa54af1e63f14ed17725f0556daf7c1b77aa1c4105844fee034d0f321b87" exitCode=0 Jul 08 18:50:09.830497 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:09.830479 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754045f7k" event={"ID":"e111e6ad-cb37-40db-aaa1-a5ae5cea6331","Type":"ContainerDied","Data":"5ec6aa54af1e63f14ed17725f0556daf7c1b77aa1c4105844fee034d0f321b87"} Jul 08 18:50:12.843780 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:12.843746 2570 generic.go:356] "Generic (PLEG): container finished" podID="e111e6ad-cb37-40db-aaa1-a5ae5cea6331" containerID="886649474e3b04de16e5d77f8b145cecc47e0dca65cdcf3d5f3b26c7d12a6962" exitCode=0 Jul 08 18:50:12.844224 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:12.843833 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754045f7k" event={"ID":"e111e6ad-cb37-40db-aaa1-a5ae5cea6331","Type":"ContainerDied","Data":"886649474e3b04de16e5d77f8b145cecc47e0dca65cdcf3d5f3b26c7d12a6962"} Jul 08 18:50:22.887307 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:22.887275 2570 generic.go:356] "Generic (PLEG): container finished" podID="e111e6ad-cb37-40db-aaa1-a5ae5cea6331" containerID="1bad6ce3609e9e92638f2139fe9597b8fc6bec1cc65ad65ef702bef33ea4ac6b" exitCode=0 Jul 08 18:50:22.887754 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:22.887349 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754045f7k" event={"ID":"e111e6ad-cb37-40db-aaa1-a5ae5cea6331","Type":"ContainerDied","Data":"1bad6ce3609e9e92638f2139fe9597b8fc6bec1cc65ad65ef702bef33ea4ac6b"} Jul 08 18:50:24.037396 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:24.037360 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754045f7k" Jul 08 18:50:24.131754 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:24.131713 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e111e6ad-cb37-40db-aaa1-a5ae5cea6331-util\") pod \"e111e6ad-cb37-40db-aaa1-a5ae5cea6331\" (UID: \"e111e6ad-cb37-40db-aaa1-a5ae5cea6331\") " Jul 08 18:50:24.131754 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:24.131756 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e111e6ad-cb37-40db-aaa1-a5ae5cea6331-bundle\") pod \"e111e6ad-cb37-40db-aaa1-a5ae5cea6331\" (UID: \"e111e6ad-cb37-40db-aaa1-a5ae5cea6331\") " Jul 08 18:50:24.131976 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:24.131794 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94ftv\" (UniqueName: \"kubernetes.io/projected/e111e6ad-cb37-40db-aaa1-a5ae5cea6331-kube-api-access-94ftv\") pod \"e111e6ad-cb37-40db-aaa1-a5ae5cea6331\" (UID: \"e111e6ad-cb37-40db-aaa1-a5ae5cea6331\") " Jul 08 18:50:24.132611 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:24.132574 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e111e6ad-cb37-40db-aaa1-a5ae5cea6331-bundle" (OuterVolumeSpecName: "bundle") pod "e111e6ad-cb37-40db-aaa1-a5ae5cea6331" (UID: "e111e6ad-cb37-40db-aaa1-a5ae5cea6331"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:50:24.133971 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:24.133949 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e111e6ad-cb37-40db-aaa1-a5ae5cea6331-kube-api-access-94ftv" (OuterVolumeSpecName: "kube-api-access-94ftv") pod "e111e6ad-cb37-40db-aaa1-a5ae5cea6331" (UID: "e111e6ad-cb37-40db-aaa1-a5ae5cea6331"). InnerVolumeSpecName "kube-api-access-94ftv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:50:24.138449 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:24.138425 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e111e6ad-cb37-40db-aaa1-a5ae5cea6331-util" (OuterVolumeSpecName: "util") pod "e111e6ad-cb37-40db-aaa1-a5ae5cea6331" (UID: "e111e6ad-cb37-40db-aaa1-a5ae5cea6331"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:50:24.232901 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:24.232794 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e111e6ad-cb37-40db-aaa1-a5ae5cea6331-util\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:50:24.232901 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:24.232862 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e111e6ad-cb37-40db-aaa1-a5ae5cea6331-bundle\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:50:24.232901 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:24.232874 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94ftv\" (UniqueName: \"kubernetes.io/projected/e111e6ad-cb37-40db-aaa1-a5ae5cea6331-kube-api-access-94ftv\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:50:24.896446 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:24.896416 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754045f7k" Jul 08 18:50:24.896646 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:24.896408 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754045f7k" event={"ID":"e111e6ad-cb37-40db-aaa1-a5ae5cea6331","Type":"ContainerDied","Data":"4863161d1e913cdc5df6deac0c7709957cf7a17c5477877e9f943448c4a7f575"} Jul 08 18:50:24.896646 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:24.896565 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4863161d1e913cdc5df6deac0c7709957cf7a17c5477877e9f943448c4a7f575" Jul 08 18:50:28.718675 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:28.718640 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kztrm"] Jul 08 18:50:28.719134 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:28.718911 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e111e6ad-cb37-40db-aaa1-a5ae5cea6331" containerName="util" Jul 08 18:50:28.719134 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:28.718923 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e111e6ad-cb37-40db-aaa1-a5ae5cea6331" containerName="util" Jul 08 18:50:28.719134 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:28.718935 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e111e6ad-cb37-40db-aaa1-a5ae5cea6331" containerName="extract" Jul 08 18:50:28.719134 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:28.718941 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e111e6ad-cb37-40db-aaa1-a5ae5cea6331" containerName="extract" Jul 08 18:50:28.719134 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:28.718952 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e111e6ad-cb37-40db-aaa1-a5ae5cea6331" containerName="pull" Jul 08 18:50:28.719134 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:28.718958 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e111e6ad-cb37-40db-aaa1-a5ae5cea6331" containerName="pull" Jul 08 18:50:28.719134 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:28.719009 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e111e6ad-cb37-40db-aaa1-a5ae5cea6331" containerName="extract" Jul 08 18:50:28.720954 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:28.720937 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kztrm" Jul 08 18:50:28.725574 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:28.725552 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-z8vqx\"" Jul 08 18:50:28.725714 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:28.725594 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 08 18:50:28.725869 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:28.725848 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 08 18:50:28.737202 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:28.737181 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kztrm"] Jul 08 18:50:28.771732 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:28.771703 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbce0e06-0f77-4032-8ccb-784234640c54-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-kztrm\" (UID: \"cbce0e06-0f77-4032-8ccb-784234640c54\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kztrm" Jul 08 18:50:28.771887 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:28.771762 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgts2\" (UniqueName: \"kubernetes.io/projected/cbce0e06-0f77-4032-8ccb-784234640c54-kube-api-access-hgts2\") pod \"cert-manager-operator-controller-manager-55c444bf86-kztrm\" (UID: \"cbce0e06-0f77-4032-8ccb-784234640c54\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kztrm" Jul 08 18:50:28.873054 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:28.873016 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbce0e06-0f77-4032-8ccb-784234640c54-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-kztrm\" (UID: \"cbce0e06-0f77-4032-8ccb-784234640c54\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kztrm" Jul 08 18:50:28.873195 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:28.873076 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgts2\" (UniqueName: \"kubernetes.io/projected/cbce0e06-0f77-4032-8ccb-784234640c54-kube-api-access-hgts2\") pod \"cert-manager-operator-controller-manager-55c444bf86-kztrm\" (UID: \"cbce0e06-0f77-4032-8ccb-784234640c54\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kztrm" Jul 08 18:50:28.873434 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:28.873413 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbce0e06-0f77-4032-8ccb-784234640c54-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-kztrm\" (UID: \"cbce0e06-0f77-4032-8ccb-784234640c54\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kztrm" Jul 08 18:50:28.885199 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:28.885165 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgts2\" (UniqueName: \"kubernetes.io/projected/cbce0e06-0f77-4032-8ccb-784234640c54-kube-api-access-hgts2\") pod \"cert-manager-operator-controller-manager-55c444bf86-kztrm\" (UID: \"cbce0e06-0f77-4032-8ccb-784234640c54\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kztrm" Jul 08 18:50:29.029802 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:29.029714 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kztrm" Jul 08 18:50:29.164349 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:29.164317 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kztrm"] Jul 08 18:50:29.167782 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:50:29.167755 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbce0e06_0f77_4032_8ccb_784234640c54.slice/crio-6d56bbddaca993920e9b3d8cdd57cc8a5a19e9fc2a25d129171e2cb5f9bf3043 WatchSource:0}: Error finding container 6d56bbddaca993920e9b3d8cdd57cc8a5a19e9fc2a25d129171e2cb5f9bf3043: Status 404 returned error can't find the container with id 6d56bbddaca993920e9b3d8cdd57cc8a5a19e9fc2a25d129171e2cb5f9bf3043 Jul 08 18:50:29.914723 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:29.914675 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kztrm" event={"ID":"cbce0e06-0f77-4032-8ccb-784234640c54","Type":"ContainerStarted","Data":"6d56bbddaca993920e9b3d8cdd57cc8a5a19e9fc2a25d129171e2cb5f9bf3043"} Jul 08 18:50:31.925702 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:31.925667 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kztrm" event={"ID":"cbce0e06-0f77-4032-8ccb-784234640c54","Type":"ContainerStarted","Data":"fe776e0992e5a3d88670663fb66e2c060b6e3c3a54bcac5c6898ad81d9aa3272"} Jul 08 18:50:31.950716 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:31.950664 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kztrm" podStartSLOduration=1.910666602 podStartE2EDuration="3.950646338s" totalImagesPullingTime="2.039979736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:50:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:50:31.948082482 +0000 UTC m=+292.795357161" watchObservedRunningTime="2026-07-08 18:50:31.950646338 +0000 UTC m=+292.797921004" Jul 08 18:50:32.937750 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:32.937719 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhfqwh"] Jul 08 18:50:32.940982 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:32.940967 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhfqwh" Jul 08 18:50:32.944094 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:32.944067 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rftlb\"" Jul 08 18:50:32.945456 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:32.945428 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 08 18:50:32.945456 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:32.945447 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 08 18:50:32.950705 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:32.950685 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhfqwh"] Jul 08 18:50:33.006124 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:33.006089 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b45a015-391e-4f2a-aed6-0d64a440530a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhfqwh\" (UID: \"3b45a015-391e-4f2a-aed6-0d64a440530a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhfqwh" Jul 08 18:50:33.006322 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:33.006139 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg67s\" (UniqueName: \"kubernetes.io/projected/3b45a015-391e-4f2a-aed6-0d64a440530a-kube-api-access-xg67s\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhfqwh\" (UID: \"3b45a015-391e-4f2a-aed6-0d64a440530a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhfqwh" Jul 08 18:50:33.006322 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:33.006203 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b45a015-391e-4f2a-aed6-0d64a440530a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhfqwh\" (UID: \"3b45a015-391e-4f2a-aed6-0d64a440530a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhfqwh" Jul 08 18:50:33.106853 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:33.106814 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b45a015-391e-4f2a-aed6-0d64a440530a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhfqwh\" (UID: \"3b45a015-391e-4f2a-aed6-0d64a440530a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhfqwh" Jul 08 18:50:33.107049 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:33.106916 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b45a015-391e-4f2a-aed6-0d64a440530a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhfqwh\" (UID: \"3b45a015-391e-4f2a-aed6-0d64a440530a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhfqwh" Jul 08 18:50:33.107049 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:33.106953 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg67s\" (UniqueName: \"kubernetes.io/projected/3b45a015-391e-4f2a-aed6-0d64a440530a-kube-api-access-xg67s\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhfqwh\" (UID: \"3b45a015-391e-4f2a-aed6-0d64a440530a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhfqwh" Jul 08 18:50:33.107317 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:33.107293 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b45a015-391e-4f2a-aed6-0d64a440530a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhfqwh\" (UID: \"3b45a015-391e-4f2a-aed6-0d64a440530a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhfqwh" Jul 08 18:50:33.107396 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:33.107312 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b45a015-391e-4f2a-aed6-0d64a440530a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhfqwh\" (UID: \"3b45a015-391e-4f2a-aed6-0d64a440530a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhfqwh" Jul 08 18:50:33.118981 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:33.118952 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg67s\" (UniqueName: \"kubernetes.io/projected/3b45a015-391e-4f2a-aed6-0d64a440530a-kube-api-access-xg67s\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhfqwh\" (UID: \"3b45a015-391e-4f2a-aed6-0d64a440530a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhfqwh" Jul 08 18:50:33.250572 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:33.250489 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhfqwh" Jul 08 18:50:33.382679 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:33.382653 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhfqwh"] Jul 08 18:50:33.384937 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:50:33.384911 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b45a015_391e_4f2a_aed6_0d64a440530a.slice/crio-a156eaea67dd2ec0cffc01ee0cd3d9a02dae9ca2f69a6e538ec7870faf6a3344 WatchSource:0}: Error finding container a156eaea67dd2ec0cffc01ee0cd3d9a02dae9ca2f69a6e538ec7870faf6a3344: Status 404 returned error can't find the container with id a156eaea67dd2ec0cffc01ee0cd3d9a02dae9ca2f69a6e538ec7870faf6a3344 Jul 08 18:50:33.935690 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:33.935657 2570 generic.go:356] "Generic (PLEG): container finished" podID="3b45a015-391e-4f2a-aed6-0d64a440530a" containerID="c9bcce405bd22c17ac979f5bf1a901b19b5dfcbb5abc1b1154045821a117653e" exitCode=0 Jul 08 18:50:33.935838 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:33.935717 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhfqwh" event={"ID":"3b45a015-391e-4f2a-aed6-0d64a440530a","Type":"ContainerDied","Data":"c9bcce405bd22c17ac979f5bf1a901b19b5dfcbb5abc1b1154045821a117653e"} Jul 08 18:50:33.935838 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:33.935752 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhfqwh" event={"ID":"3b45a015-391e-4f2a-aed6-0d64a440530a","Type":"ContainerStarted","Data":"a156eaea67dd2ec0cffc01ee0cd3d9a02dae9ca2f69a6e538ec7870faf6a3344"} Jul 08 18:50:36.951086 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:36.951049 2570 generic.go:356] "Generic (PLEG): container finished" podID="3b45a015-391e-4f2a-aed6-0d64a440530a" containerID="c0cc969b66c8c39ff012f74f278ae16a09f53cbf33785f7523266abcd4f24181" exitCode=0 Jul 08 18:50:36.951489 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:36.951100 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhfqwh" event={"ID":"3b45a015-391e-4f2a-aed6-0d64a440530a","Type":"ContainerDied","Data":"c0cc969b66c8c39ff012f74f278ae16a09f53cbf33785f7523266abcd4f24181"} Jul 08 18:50:37.957902 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:37.957866 2570 generic.go:356] "Generic (PLEG): container finished" podID="3b45a015-391e-4f2a-aed6-0d64a440530a" containerID="f0288d8a82191621421386b869ac84ce0d704adabed2dd054d92a245797f0860" exitCode=0 Jul 08 18:50:37.958326 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:37.957954 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhfqwh" event={"ID":"3b45a015-391e-4f2a-aed6-0d64a440530a","Type":"ContainerDied","Data":"f0288d8a82191621421386b869ac84ce0d704adabed2dd054d92a245797f0860"} Jul 08 18:50:38.771006 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:38.770974 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-rvdfr"] Jul 08 18:50:38.774360 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:38.774342 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rvdfr" Jul 08 18:50:38.776901 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:38.776882 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-xdcmw\"" Jul 08 18:50:38.777018 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:38.776961 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 08 18:50:38.777908 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:38.777894 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 08 18:50:38.782067 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:38.782045 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-rvdfr"] Jul 08 18:50:38.859141 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:38.859095 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65rk5\" (UniqueName: \"kubernetes.io/projected/e248e6af-96e9-44ce-a322-d0e0cceee070-kube-api-access-65rk5\") pod \"cert-manager-cainjector-559cdd8c87-rvdfr\" (UID: \"e248e6af-96e9-44ce-a322-d0e0cceee070\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rvdfr" Jul 08 18:50:38.859331 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:38.859234 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e248e6af-96e9-44ce-a322-d0e0cceee070-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-rvdfr\" (UID: \"e248e6af-96e9-44ce-a322-d0e0cceee070\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rvdfr" Jul 08 18:50:38.959618 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:38.959584 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e248e6af-96e9-44ce-a322-d0e0cceee070-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-rvdfr\" (UID: \"e248e6af-96e9-44ce-a322-d0e0cceee070\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rvdfr" Jul 08 18:50:38.960029 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:38.959627 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65rk5\" (UniqueName: \"kubernetes.io/projected/e248e6af-96e9-44ce-a322-d0e0cceee070-kube-api-access-65rk5\") pod \"cert-manager-cainjector-559cdd8c87-rvdfr\" (UID: \"e248e6af-96e9-44ce-a322-d0e0cceee070\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rvdfr" Jul 08 18:50:38.974717 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:38.974683 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e248e6af-96e9-44ce-a322-d0e0cceee070-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-rvdfr\" (UID: \"e248e6af-96e9-44ce-a322-d0e0cceee070\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rvdfr" Jul 08 18:50:38.974882 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:38.974692 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65rk5\" (UniqueName: \"kubernetes.io/projected/e248e6af-96e9-44ce-a322-d0e0cceee070-kube-api-access-65rk5\") pod \"cert-manager-cainjector-559cdd8c87-rvdfr\" (UID: \"e248e6af-96e9-44ce-a322-d0e0cceee070\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rvdfr" Jul 08 18:50:39.090099 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:39.090073 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhfqwh" Jul 08 18:50:39.098899 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:39.098880 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rvdfr" Jul 08 18:50:39.161836 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:39.161800 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b45a015-391e-4f2a-aed6-0d64a440530a-util\") pod \"3b45a015-391e-4f2a-aed6-0d64a440530a\" (UID: \"3b45a015-391e-4f2a-aed6-0d64a440530a\") " Jul 08 18:50:39.162003 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:39.161865 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xg67s\" (UniqueName: \"kubernetes.io/projected/3b45a015-391e-4f2a-aed6-0d64a440530a-kube-api-access-xg67s\") pod \"3b45a015-391e-4f2a-aed6-0d64a440530a\" (UID: \"3b45a015-391e-4f2a-aed6-0d64a440530a\") " Jul 08 18:50:39.162003 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:39.161906 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b45a015-391e-4f2a-aed6-0d64a440530a-bundle\") pod \"3b45a015-391e-4f2a-aed6-0d64a440530a\" (UID: \"3b45a015-391e-4f2a-aed6-0d64a440530a\") " Jul 08 18:50:39.162400 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:39.162370 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b45a015-391e-4f2a-aed6-0d64a440530a-bundle" (OuterVolumeSpecName: "bundle") pod "3b45a015-391e-4f2a-aed6-0d64a440530a" (UID: "3b45a015-391e-4f2a-aed6-0d64a440530a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:50:39.164460 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:39.164427 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b45a015-391e-4f2a-aed6-0d64a440530a-kube-api-access-xg67s" (OuterVolumeSpecName: "kube-api-access-xg67s") pod "3b45a015-391e-4f2a-aed6-0d64a440530a" (UID: "3b45a015-391e-4f2a-aed6-0d64a440530a"). InnerVolumeSpecName "kube-api-access-xg67s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:50:39.168469 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:39.168441 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b45a015-391e-4f2a-aed6-0d64a440530a-util" (OuterVolumeSpecName: "util") pod "3b45a015-391e-4f2a-aed6-0d64a440530a" (UID: "3b45a015-391e-4f2a-aed6-0d64a440530a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:50:39.231907 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:39.231882 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-rvdfr"] Jul 08 18:50:39.234355 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:50:39.234329 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode248e6af_96e9_44ce_a322_d0e0cceee070.slice/crio-22103d3db34fb93a56784efab761479306cc9cd658346240433ca5996eddbe73 WatchSource:0}: Error finding container 22103d3db34fb93a56784efab761479306cc9cd658346240433ca5996eddbe73: Status 404 returned error can't find the container with id 22103d3db34fb93a56784efab761479306cc9cd658346240433ca5996eddbe73 Jul 08 18:50:39.263219 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:39.263192 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b45a015-391e-4f2a-aed6-0d64a440530a-util\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:50:39.263219 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:39.263215 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg67s\" (UniqueName: \"kubernetes.io/projected/3b45a015-391e-4f2a-aed6-0d64a440530a-kube-api-access-xg67s\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:50:39.263391 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:39.263226 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b45a015-391e-4f2a-aed6-0d64a440530a-bundle\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:50:39.536205 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:39.536177 2570 kubelet.go:1648] "Image garbage collection succeeded" Jul 08 18:50:39.967469 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:39.967436 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhfqwh" event={"ID":"3b45a015-391e-4f2a-aed6-0d64a440530a","Type":"ContainerDied","Data":"a156eaea67dd2ec0cffc01ee0cd3d9a02dae9ca2f69a6e538ec7870faf6a3344"} Jul 08 18:50:39.967972 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:39.967488 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a156eaea67dd2ec0cffc01ee0cd3d9a02dae9ca2f69a6e538ec7870faf6a3344" Jul 08 18:50:39.967972 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:39.967444 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhfqwh" Jul 08 18:50:39.968721 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:39.968699 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rvdfr" event={"ID":"e248e6af-96e9-44ce-a322-d0e0cceee070","Type":"ContainerStarted","Data":"22103d3db34fb93a56784efab761479306cc9cd658346240433ca5996eddbe73"} Jul 08 18:50:42.983314 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:42.983272 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rvdfr" event={"ID":"e248e6af-96e9-44ce-a322-d0e0cceee070","Type":"ContainerStarted","Data":"38ac1d1131b4c14fdbf9d813be8d6acecc1c549af57d7e73f089076ea32bd3f9"} Jul 08 18:50:43.001362 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:43.001308 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rvdfr" podStartSLOduration=2.033012201 podStartE2EDuration="5.001293311s" totalImagesPullingTime="2.96828111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:50:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:50:42.999398443 +0000 UTC m=+303.846673108" watchObservedRunningTime="2026-07-08 18:50:43.001293311 +0000 UTC m=+303.848567985" Jul 08 18:50:46.433242 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:46.433206 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pk85g"] Jul 08 18:50:46.433638 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:46.433507 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b45a015-391e-4f2a-aed6-0d64a440530a" containerName="extract" Jul 08 18:50:46.433638 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:46.433519 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b45a015-391e-4f2a-aed6-0d64a440530a" containerName="extract" Jul 08 18:50:46.433638 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:46.433533 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b45a015-391e-4f2a-aed6-0d64a440530a" containerName="util" Jul 08 18:50:46.433638 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:46.433540 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b45a015-391e-4f2a-aed6-0d64a440530a" containerName="util" Jul 08 18:50:46.433638 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:46.433550 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b45a015-391e-4f2a-aed6-0d64a440530a" containerName="pull" Jul 08 18:50:46.433638 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:46.433555 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b45a015-391e-4f2a-aed6-0d64a440530a" containerName="pull" Jul 08 18:50:46.433638 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:46.433608 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b45a015-391e-4f2a-aed6-0d64a440530a" containerName="extract" Jul 08 18:50:46.437183 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:46.437165 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pk85g" Jul 08 18:50:46.439925 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:46.439893 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 08 18:50:46.440077 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:46.439945 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-g9b7b\"" Jul 08 18:50:46.440077 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:46.439949 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 08 18:50:46.448290 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:46.448266 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pk85g"] Jul 08 18:50:46.528637 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:46.528602 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9bds\" (UniqueName: \"kubernetes.io/projected/78c3ef05-a6b9-4eed-89f5-8618a93bb2d8-kube-api-access-f9bds\") pod \"openshift-lws-operator-fd8ccff4c-pk85g\" (UID: \"78c3ef05-a6b9-4eed-89f5-8618a93bb2d8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pk85g" Jul 08 18:50:46.528827 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:46.528678 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78c3ef05-a6b9-4eed-89f5-8618a93bb2d8-tmp\") pod \"openshift-lws-operator-fd8ccff4c-pk85g\" (UID: \"78c3ef05-a6b9-4eed-89f5-8618a93bb2d8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pk85g" Jul 08 18:50:46.630054 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:46.630027 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78c3ef05-a6b9-4eed-89f5-8618a93bb2d8-tmp\") pod \"openshift-lws-operator-fd8ccff4c-pk85g\" (UID: \"78c3ef05-a6b9-4eed-89f5-8618a93bb2d8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pk85g" Jul 08 18:50:46.630222 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:46.630069 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9bds\" (UniqueName: \"kubernetes.io/projected/78c3ef05-a6b9-4eed-89f5-8618a93bb2d8-kube-api-access-f9bds\") pod \"openshift-lws-operator-fd8ccff4c-pk85g\" (UID: \"78c3ef05-a6b9-4eed-89f5-8618a93bb2d8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pk85g" Jul 08 18:50:46.630490 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:46.630469 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78c3ef05-a6b9-4eed-89f5-8618a93bb2d8-tmp\") pod \"openshift-lws-operator-fd8ccff4c-pk85g\" (UID: \"78c3ef05-a6b9-4eed-89f5-8618a93bb2d8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pk85g" Jul 08 18:50:46.638936 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:46.638905 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9bds\" (UniqueName: \"kubernetes.io/projected/78c3ef05-a6b9-4eed-89f5-8618a93bb2d8-kube-api-access-f9bds\") pod \"openshift-lws-operator-fd8ccff4c-pk85g\" (UID: \"78c3ef05-a6b9-4eed-89f5-8618a93bb2d8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pk85g" Jul 08 18:50:46.746123 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:46.746032 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pk85g" Jul 08 18:50:46.893410 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:46.893376 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pk85g"] Jul 08 18:50:46.896355 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:50:46.896327 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78c3ef05_a6b9_4eed_89f5_8618a93bb2d8.slice/crio-a56278d64a9698b60cf1c17db7f4e373f8dc1966b3d0d648229c22f6918197d2 WatchSource:0}: Error finding container a56278d64a9698b60cf1c17db7f4e373f8dc1966b3d0d648229c22f6918197d2: Status 404 returned error can't find the container with id a56278d64a9698b60cf1c17db7f4e373f8dc1966b3d0d648229c22f6918197d2 Jul 08 18:50:46.897837 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:46.897821 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 18:50:46.999600 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:46.999507 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pk85g" event={"ID":"78c3ef05-a6b9-4eed-89f5-8618a93bb2d8","Type":"ContainerStarted","Data":"a56278d64a9698b60cf1c17db7f4e373f8dc1966b3d0d648229c22f6918197d2"} Jul 08 18:50:50.014237 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:50.014201 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pk85g" event={"ID":"78c3ef05-a6b9-4eed-89f5-8618a93bb2d8","Type":"ContainerStarted","Data":"a17f7249c1e3787084d34da3616d97ceb1841581c6830c395a352ffee9ccb865"} Jul 08 18:50:52.679918 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:52.679865 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pk85g" podStartSLOduration=4.416541796 podStartE2EDuration="6.679848912s" totalImagesPullingTime="2.263307116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:50:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:50:50.030448544 +0000 UTC m=+310.877723210" watchObservedRunningTime="2026-07-08 18:50:52.679848912 +0000 UTC m=+313.527123577" Jul 08 18:50:52.681230 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:52.681209 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnch4"] Jul 08 18:50:52.729455 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:52.729426 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnch4"] Jul 08 18:50:52.729598 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:52.729527 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnch4" Jul 08 18:50:52.732114 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:52.732092 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rftlb\"" Jul 08 18:50:52.732242 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:52.732159 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 08 18:50:52.732319 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:52.732281 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 08 18:50:52.880476 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:52.880436 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/687551fc-22e6-4afb-ae82-15ad8db8291b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnch4\" (UID: \"687551fc-22e6-4afb-ae82-15ad8db8291b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnch4" Jul 08 18:50:52.880655 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:52.880488 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/687551fc-22e6-4afb-ae82-15ad8db8291b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnch4\" (UID: \"687551fc-22e6-4afb-ae82-15ad8db8291b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnch4" Jul 08 18:50:52.880655 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:52.880592 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m6jc\" (UniqueName: \"kubernetes.io/projected/687551fc-22e6-4afb-ae82-15ad8db8291b-kube-api-access-8m6jc\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnch4\" (UID: \"687551fc-22e6-4afb-ae82-15ad8db8291b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnch4" Jul 08 18:50:52.981781 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:52.981702 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m6jc\" (UniqueName: \"kubernetes.io/projected/687551fc-22e6-4afb-ae82-15ad8db8291b-kube-api-access-8m6jc\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnch4\" (UID: \"687551fc-22e6-4afb-ae82-15ad8db8291b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnch4" Jul 08 18:50:52.981781 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:52.981758 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/687551fc-22e6-4afb-ae82-15ad8db8291b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnch4\" (UID: \"687551fc-22e6-4afb-ae82-15ad8db8291b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnch4" Jul 08 18:50:52.981985 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:52.981786 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/687551fc-22e6-4afb-ae82-15ad8db8291b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnch4\" (UID: \"687551fc-22e6-4afb-ae82-15ad8db8291b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnch4" Jul 08 18:50:52.982098 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:52.982081 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/687551fc-22e6-4afb-ae82-15ad8db8291b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnch4\" (UID: \"687551fc-22e6-4afb-ae82-15ad8db8291b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnch4" Jul 08 18:50:52.982150 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:52.982132 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/687551fc-22e6-4afb-ae82-15ad8db8291b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnch4\" (UID: \"687551fc-22e6-4afb-ae82-15ad8db8291b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnch4" Jul 08 18:50:52.992133 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:52.992102 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m6jc\" (UniqueName: \"kubernetes.io/projected/687551fc-22e6-4afb-ae82-15ad8db8291b-kube-api-access-8m6jc\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnch4\" (UID: \"687551fc-22e6-4afb-ae82-15ad8db8291b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnch4" Jul 08 18:50:53.038881 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:53.038859 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnch4" Jul 08 18:50:53.205976 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:53.205935 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnch4"] Jul 08 18:50:53.207484 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:50:53.207455 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod687551fc_22e6_4afb_ae82_15ad8db8291b.slice/crio-93e141754e7352e7c61ade5048ea503234c94a6abde65f892812c9144eaf8c45 WatchSource:0}: Error finding container 93e141754e7352e7c61ade5048ea503234c94a6abde65f892812c9144eaf8c45: Status 404 returned error can't find the container with id 93e141754e7352e7c61ade5048ea503234c94a6abde65f892812c9144eaf8c45 Jul 08 18:50:54.031585 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:54.031543 2570 generic.go:356] "Generic (PLEG): container finished" podID="687551fc-22e6-4afb-ae82-15ad8db8291b" containerID="c2d79fe179a54abe7461d3f2d277acf0209545fec4ce7138096319fba2725c44" exitCode=0 Jul 08 18:50:54.031923 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:54.031625 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnch4" event={"ID":"687551fc-22e6-4afb-ae82-15ad8db8291b","Type":"ContainerDied","Data":"c2d79fe179a54abe7461d3f2d277acf0209545fec4ce7138096319fba2725c44"} Jul 08 18:50:54.031923 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:54.031672 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnch4" event={"ID":"687551fc-22e6-4afb-ae82-15ad8db8291b","Type":"ContainerStarted","Data":"93e141754e7352e7c61ade5048ea503234c94a6abde65f892812c9144eaf8c45"} Jul 08 18:50:56.041967 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:56.041935 2570 generic.go:356] "Generic (PLEG): container finished" podID="687551fc-22e6-4afb-ae82-15ad8db8291b" containerID="6976b6441f91735f49191cd7cc72375f563e44df11f7676882bd65e014370d41" exitCode=0 Jul 08 18:50:56.042472 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:56.042013 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnch4" event={"ID":"687551fc-22e6-4afb-ae82-15ad8db8291b","Type":"ContainerDied","Data":"6976b6441f91735f49191cd7cc72375f563e44df11f7676882bd65e014370d41"} Jul 08 18:50:57.048902 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:57.048868 2570 generic.go:356] "Generic (PLEG): container finished" podID="687551fc-22e6-4afb-ae82-15ad8db8291b" containerID="7164ad3e122121c1c042b8ba2f5aea7255223c4b9c93f0a4efd2ad15ae68a4c6" exitCode=0 Jul 08 18:50:57.049327 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:57.048956 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnch4" event={"ID":"687551fc-22e6-4afb-ae82-15ad8db8291b","Type":"ContainerDied","Data":"7164ad3e122121c1c042b8ba2f5aea7255223c4b9c93f0a4efd2ad15ae68a4c6"} Jul 08 18:50:58.176277 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:58.176232 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnch4" Jul 08 18:50:58.222300 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:58.222278 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8m6jc\" (UniqueName: \"kubernetes.io/projected/687551fc-22e6-4afb-ae82-15ad8db8291b-kube-api-access-8m6jc\") pod \"687551fc-22e6-4afb-ae82-15ad8db8291b\" (UID: \"687551fc-22e6-4afb-ae82-15ad8db8291b\") " Jul 08 18:50:58.222422 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:58.222308 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/687551fc-22e6-4afb-ae82-15ad8db8291b-bundle\") pod \"687551fc-22e6-4afb-ae82-15ad8db8291b\" (UID: \"687551fc-22e6-4afb-ae82-15ad8db8291b\") " Jul 08 18:50:58.222422 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:58.222342 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/687551fc-22e6-4afb-ae82-15ad8db8291b-util\") pod \"687551fc-22e6-4afb-ae82-15ad8db8291b\" (UID: \"687551fc-22e6-4afb-ae82-15ad8db8291b\") " Jul 08 18:50:58.222961 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:58.222933 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687551fc-22e6-4afb-ae82-15ad8db8291b-bundle" (OuterVolumeSpecName: "bundle") pod "687551fc-22e6-4afb-ae82-15ad8db8291b" (UID: "687551fc-22e6-4afb-ae82-15ad8db8291b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:50:58.224303 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:58.224283 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/687551fc-22e6-4afb-ae82-15ad8db8291b-kube-api-access-8m6jc" (OuterVolumeSpecName: "kube-api-access-8m6jc") pod "687551fc-22e6-4afb-ae82-15ad8db8291b" (UID: "687551fc-22e6-4afb-ae82-15ad8db8291b"). InnerVolumeSpecName "kube-api-access-8m6jc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:50:58.230150 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:58.230122 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687551fc-22e6-4afb-ae82-15ad8db8291b-util" (OuterVolumeSpecName: "util") pod "687551fc-22e6-4afb-ae82-15ad8db8291b" (UID: "687551fc-22e6-4afb-ae82-15ad8db8291b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:50:58.323103 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:58.323047 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m6jc\" (UniqueName: \"kubernetes.io/projected/687551fc-22e6-4afb-ae82-15ad8db8291b-kube-api-access-8m6jc\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:50:58.323103 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:58.323069 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/687551fc-22e6-4afb-ae82-15ad8db8291b-bundle\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:50:58.323103 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:58.323079 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/687551fc-22e6-4afb-ae82-15ad8db8291b-util\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:50:59.059230 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:59.059187 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnch4" event={"ID":"687551fc-22e6-4afb-ae82-15ad8db8291b","Type":"ContainerDied","Data":"93e141754e7352e7c61ade5048ea503234c94a6abde65f892812c9144eaf8c45"} Jul 08 18:50:59.059230 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:59.059220 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnch4" Jul 08 18:50:59.059230 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:50:59.059239 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93e141754e7352e7c61ade5048ea503234c94a6abde65f892812c9144eaf8c45" Jul 08 18:51:09.141906 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:09.141870 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bb8746f75-2qk9m"] Jul 08 18:51:09.142399 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:09.142161 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="687551fc-22e6-4afb-ae82-15ad8db8291b" containerName="pull" Jul 08 18:51:09.142399 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:09.142172 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="687551fc-22e6-4afb-ae82-15ad8db8291b" containerName="pull" Jul 08 18:51:09.142399 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:09.142179 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="687551fc-22e6-4afb-ae82-15ad8db8291b" containerName="util" Jul 08 18:51:09.142399 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:09.142183 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="687551fc-22e6-4afb-ae82-15ad8db8291b" containerName="util" Jul 08 18:51:09.142399 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:09.142204 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="687551fc-22e6-4afb-ae82-15ad8db8291b" containerName="extract" Jul 08 18:51:09.142399 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:09.142210 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="687551fc-22e6-4afb-ae82-15ad8db8291b" containerName="extract" Jul 08 18:51:09.142399 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:09.142275 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="687551fc-22e6-4afb-ae82-15ad8db8291b" containerName="extract" Jul 08 18:51:09.144325 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:09.144308 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6bb8746f75-2qk9m" Jul 08 18:51:09.148227 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:09.148206 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 08 18:51:09.148387 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:09.148372 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-4vdkb\"" Jul 08 18:51:09.148985 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:09.148969 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 08 18:51:09.149068 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:09.149024 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 08 18:51:09.149134 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:09.149084 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 08 18:51:09.170135 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:09.170107 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bb8746f75-2qk9m"] Jul 08 18:51:09.208211 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:09.208190 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d608d49-4650-4c31-a350-bbcbe95baef2-tmp\") pod \"opendatahub-operator-controller-manager-6bb8746f75-2qk9m\" (UID: \"9d608d49-4650-4c31-a350-bbcbe95baef2\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb8746f75-2qk9m" Jul 08 18:51:09.208327 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:09.208235 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9d608d49-4650-4c31-a350-bbcbe95baef2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bb8746f75-2qk9m\" (UID: \"9d608d49-4650-4c31-a350-bbcbe95baef2\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb8746f75-2qk9m" Jul 08 18:51:09.208327 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:09.208301 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xshp\" (UniqueName: \"kubernetes.io/projected/9d608d49-4650-4c31-a350-bbcbe95baef2-kube-api-access-9xshp\") pod \"opendatahub-operator-controller-manager-6bb8746f75-2qk9m\" (UID: \"9d608d49-4650-4c31-a350-bbcbe95baef2\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb8746f75-2qk9m" Jul 08 18:51:09.208406 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:09.208377 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9d608d49-4650-4c31-a350-bbcbe95baef2-manifests\") pod \"opendatahub-operator-controller-manager-6bb8746f75-2qk9m\" (UID: \"9d608d49-4650-4c31-a350-bbcbe95baef2\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb8746f75-2qk9m" Jul 08 18:51:09.208441 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:09.208426 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d608d49-4650-4c31-a350-bbcbe95baef2-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bb8746f75-2qk9m\" (UID: \"9d608d49-4650-4c31-a350-bbcbe95baef2\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb8746f75-2qk9m" Jul 08 18:51:09.309525 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:09.309497 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9d608d49-4650-4c31-a350-bbcbe95baef2-manifests\") pod \"opendatahub-operator-controller-manager-6bb8746f75-2qk9m\" (UID: \"9d608d49-4650-4c31-a350-bbcbe95baef2\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb8746f75-2qk9m" Jul 08 18:51:09.309653 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:09.309537 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d608d49-4650-4c31-a350-bbcbe95baef2-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bb8746f75-2qk9m\" (UID: \"9d608d49-4650-4c31-a350-bbcbe95baef2\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb8746f75-2qk9m" Jul 08 18:51:09.309653 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:09.309571 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d608d49-4650-4c31-a350-bbcbe95baef2-tmp\") pod \"opendatahub-operator-controller-manager-6bb8746f75-2qk9m\" (UID: \"9d608d49-4650-4c31-a350-bbcbe95baef2\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb8746f75-2qk9m" Jul 08 18:51:09.309653 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:09.309625 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9d608d49-4650-4c31-a350-bbcbe95baef2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bb8746f75-2qk9m\" (UID: \"9d608d49-4650-4c31-a350-bbcbe95baef2\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb8746f75-2qk9m" Jul 08 18:51:09.309773 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:09.309654 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xshp\" (UniqueName: \"kubernetes.io/projected/9d608d49-4650-4c31-a350-bbcbe95baef2-kube-api-access-9xshp\") pod \"opendatahub-operator-controller-manager-6bb8746f75-2qk9m\" (UID: \"9d608d49-4650-4c31-a350-bbcbe95baef2\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb8746f75-2qk9m" Jul 08 18:51:09.309963 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:09.309943 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9d608d49-4650-4c31-a350-bbcbe95baef2-manifests\") pod \"opendatahub-operator-controller-manager-6bb8746f75-2qk9m\" (UID: \"9d608d49-4650-4c31-a350-bbcbe95baef2\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb8746f75-2qk9m" Jul 08 18:51:09.311801 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:09.311773 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d608d49-4650-4c31-a350-bbcbe95baef2-tmp\") pod \"opendatahub-operator-controller-manager-6bb8746f75-2qk9m\" (UID: \"9d608d49-4650-4c31-a350-bbcbe95baef2\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb8746f75-2qk9m" Jul 08 18:51:09.312080 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:09.312059 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d608d49-4650-4c31-a350-bbcbe95baef2-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bb8746f75-2qk9m\" (UID: \"9d608d49-4650-4c31-a350-bbcbe95baef2\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb8746f75-2qk9m" Jul 08 18:51:09.312080 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:09.312072 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9d608d49-4650-4c31-a350-bbcbe95baef2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bb8746f75-2qk9m\" (UID: \"9d608d49-4650-4c31-a350-bbcbe95baef2\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb8746f75-2qk9m" Jul 08 18:51:09.335073 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:09.335048 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xshp\" (UniqueName: \"kubernetes.io/projected/9d608d49-4650-4c31-a350-bbcbe95baef2-kube-api-access-9xshp\") pod \"opendatahub-operator-controller-manager-6bb8746f75-2qk9m\" (UID: \"9d608d49-4650-4c31-a350-bbcbe95baef2\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb8746f75-2qk9m" Jul 08 18:51:09.454595 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:09.454521 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6bb8746f75-2qk9m" Jul 08 18:51:09.599303 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:09.599240 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bb8746f75-2qk9m"] Jul 08 18:51:09.605129 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:51:09.605102 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d608d49_4650_4c31_a350_bbcbe95baef2.slice/crio-f8066d214f45c42a2d985f401c140e2020ff58f7bfa3eef1a5a2449a1c086523 WatchSource:0}: Error finding container f8066d214f45c42a2d985f401c140e2020ff58f7bfa3eef1a5a2449a1c086523: Status 404 returned error can't find the container with id f8066d214f45c42a2d985f401c140e2020ff58f7bfa3eef1a5a2449a1c086523 Jul 08 18:51:10.107507 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:10.107466 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bb8746f75-2qk9m" event={"ID":"9d608d49-4650-4c31-a350-bbcbe95baef2","Type":"ContainerStarted","Data":"f8066d214f45c42a2d985f401c140e2020ff58f7bfa3eef1a5a2449a1c086523"} Jul 08 18:51:13.124644 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:13.124607 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bb8746f75-2qk9m" event={"ID":"9d608d49-4650-4c31-a350-bbcbe95baef2","Type":"ContainerStarted","Data":"dee15f05b5149f7bbc710bfbbdbc777ff5d8132d49dc7cbed95ef7922e97c75d"} Jul 08 18:51:14.130993 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:14.130959 2570 generic.go:356] "Generic (PLEG): container finished" podID="9d608d49-4650-4c31-a350-bbcbe95baef2" containerID="dee15f05b5149f7bbc710bfbbdbc777ff5d8132d49dc7cbed95ef7922e97c75d" exitCode=0 Jul 08 18:51:14.131402 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:14.131014 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bb8746f75-2qk9m" event={"ID":"9d608d49-4650-4c31-a350-bbcbe95baef2","Type":"ContainerDied","Data":"dee15f05b5149f7bbc710bfbbdbc777ff5d8132d49dc7cbed95ef7922e97c75d"} Jul 08 18:51:15.137582 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:15.137543 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bb8746f75-2qk9m" event={"ID":"9d608d49-4650-4c31-a350-bbcbe95baef2","Type":"ContainerStarted","Data":"bb1d91fedbe5e275d8c08fb9026e9a2c3be80484da446e0309c424bf4fec87d4"} Jul 08 18:51:15.138055 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:15.137681 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6bb8746f75-2qk9m" Jul 08 18:51:15.162686 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:15.162635 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6bb8746f75-2qk9m" podStartSLOduration=1.753322228 podStartE2EDuration="6.162618475s" totalImagesPullingTime="3.36091327s" totalInitContainerRuntime="1.048382977s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:51:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:51:15.161484056 +0000 UTC m=+336.008758723" watchObservedRunningTime="2026-07-08 18:51:15.162618475 +0000 UTC m=+336.009893141" Jul 08 18:51:26.144679 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:26.144647 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6bb8746f75-2qk9m" Jul 08 18:51:28.859524 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:28.859486 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-64798db975-sccdk"] Jul 08 18:51:28.870487 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:28.870459 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-64798db975-sccdk" Jul 08 18:51:28.875279 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:28.875232 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 08 18:51:28.876210 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:28.876008 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-n9mfr\"" Jul 08 18:51:28.876210 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:28.876017 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 08 18:51:28.876427 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:28.876210 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 08 18:51:28.876427 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:28.876230 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 08 18:51:28.882865 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:28.882843 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-64798db975-sccdk"] Jul 08 18:51:28.963905 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:28.963871 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85943ce7-bf4a-45e9-9342-36d6aae74b29-tls-certs\") pod \"kube-auth-proxy-64798db975-sccdk\" (UID: \"85943ce7-bf4a-45e9-9342-36d6aae74b29\") " pod="openshift-ingress/kube-auth-proxy-64798db975-sccdk" Jul 08 18:51:28.964097 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:28.963935 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9pzz\" (UniqueName: \"kubernetes.io/projected/85943ce7-bf4a-45e9-9342-36d6aae74b29-kube-api-access-f9pzz\") pod \"kube-auth-proxy-64798db975-sccdk\" (UID: \"85943ce7-bf4a-45e9-9342-36d6aae74b29\") " pod="openshift-ingress/kube-auth-proxy-64798db975-sccdk" Jul 08 18:51:28.964097 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:28.963979 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85943ce7-bf4a-45e9-9342-36d6aae74b29-tmp\") pod \"kube-auth-proxy-64798db975-sccdk\" (UID: \"85943ce7-bf4a-45e9-9342-36d6aae74b29\") " pod="openshift-ingress/kube-auth-proxy-64798db975-sccdk" Jul 08 18:51:29.064798 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:29.064760 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85943ce7-bf4a-45e9-9342-36d6aae74b29-tls-certs\") pod \"kube-auth-proxy-64798db975-sccdk\" (UID: \"85943ce7-bf4a-45e9-9342-36d6aae74b29\") " pod="openshift-ingress/kube-auth-proxy-64798db975-sccdk" Jul 08 18:51:29.064995 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:51:29.064904 2570 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 08 18:51:29.064995 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:29.064909 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9pzz\" (UniqueName: \"kubernetes.io/projected/85943ce7-bf4a-45e9-9342-36d6aae74b29-kube-api-access-f9pzz\") pod \"kube-auth-proxy-64798db975-sccdk\" (UID: \"85943ce7-bf4a-45e9-9342-36d6aae74b29\") " pod="openshift-ingress/kube-auth-proxy-64798db975-sccdk" Jul 08 18:51:29.064995 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:29.064954 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85943ce7-bf4a-45e9-9342-36d6aae74b29-tmp\") pod \"kube-auth-proxy-64798db975-sccdk\" (UID: \"85943ce7-bf4a-45e9-9342-36d6aae74b29\") " pod="openshift-ingress/kube-auth-proxy-64798db975-sccdk" Jul 08 18:51:29.064995 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:51:29.064974 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85943ce7-bf4a-45e9-9342-36d6aae74b29-tls-certs podName:85943ce7-bf4a-45e9-9342-36d6aae74b29 nodeName:}" failed. No retries permitted until 2026-07-08 18:51:29.564957393 +0000 UTC m=+350.412232039 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/85943ce7-bf4a-45e9-9342-36d6aae74b29-tls-certs") pod "kube-auth-proxy-64798db975-sccdk" (UID: "85943ce7-bf4a-45e9-9342-36d6aae74b29") : secret "kube-auth-proxy-tls" not found Jul 08 18:51:29.067223 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:29.067197 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85943ce7-bf4a-45e9-9342-36d6aae74b29-tmp\") pod \"kube-auth-proxy-64798db975-sccdk\" (UID: \"85943ce7-bf4a-45e9-9342-36d6aae74b29\") " pod="openshift-ingress/kube-auth-proxy-64798db975-sccdk" Jul 08 18:51:29.081821 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:29.081795 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9pzz\" (UniqueName: \"kubernetes.io/projected/85943ce7-bf4a-45e9-9342-36d6aae74b29-kube-api-access-f9pzz\") pod \"kube-auth-proxy-64798db975-sccdk\" (UID: \"85943ce7-bf4a-45e9-9342-36d6aae74b29\") " pod="openshift-ingress/kube-auth-proxy-64798db975-sccdk" Jul 08 18:51:29.135232 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:29.135153 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9xllh"] Jul 08 18:51:29.137515 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:29.137497 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-9xllh" Jul 08 18:51:29.144018 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:29.143982 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 08 18:51:29.144463 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:29.144365 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-wp2w4\"" Jul 08 18:51:29.175608 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:29.175562 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9xllh"] Jul 08 18:51:29.266533 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:29.266504 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v96zp\" (UniqueName: \"kubernetes.io/projected/df7227ff-4ae1-4f9c-8903-860139d8a70e-kube-api-access-v96zp\") pod \"odh-model-controller-7998c67dc4-9xllh\" (UID: \"df7227ff-4ae1-4f9c-8903-860139d8a70e\") " pod="opendatahub/odh-model-controller-7998c67dc4-9xllh" Jul 08 18:51:29.266706 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:29.266558 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/df7227ff-4ae1-4f9c-8903-860139d8a70e-cert\") pod \"odh-model-controller-7998c67dc4-9xllh\" (UID: \"df7227ff-4ae1-4f9c-8903-860139d8a70e\") " pod="opendatahub/odh-model-controller-7998c67dc4-9xllh" Jul 08 18:51:29.367096 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:29.367059 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/df7227ff-4ae1-4f9c-8903-860139d8a70e-cert\") pod \"odh-model-controller-7998c67dc4-9xllh\" (UID: \"df7227ff-4ae1-4f9c-8903-860139d8a70e\") " pod="opendatahub/odh-model-controller-7998c67dc4-9xllh" Jul 08 18:51:29.367306 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:29.367162 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v96zp\" (UniqueName: \"kubernetes.io/projected/df7227ff-4ae1-4f9c-8903-860139d8a70e-kube-api-access-v96zp\") pod \"odh-model-controller-7998c67dc4-9xllh\" (UID: \"df7227ff-4ae1-4f9c-8903-860139d8a70e\") " pod="opendatahub/odh-model-controller-7998c67dc4-9xllh" Jul 08 18:51:29.369634 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:29.369590 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/df7227ff-4ae1-4f9c-8903-860139d8a70e-cert\") pod \"odh-model-controller-7998c67dc4-9xllh\" (UID: \"df7227ff-4ae1-4f9c-8903-860139d8a70e\") " pod="opendatahub/odh-model-controller-7998c67dc4-9xllh" Jul 08 18:51:29.383005 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:29.382974 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v96zp\" (UniqueName: \"kubernetes.io/projected/df7227ff-4ae1-4f9c-8903-860139d8a70e-kube-api-access-v96zp\") pod \"odh-model-controller-7998c67dc4-9xllh\" (UID: \"df7227ff-4ae1-4f9c-8903-860139d8a70e\") " pod="opendatahub/odh-model-controller-7998c67dc4-9xllh" Jul 08 18:51:29.448491 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:29.448413 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-9xllh" Jul 08 18:51:29.569009 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:29.568976 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85943ce7-bf4a-45e9-9342-36d6aae74b29-tls-certs\") pod \"kube-auth-proxy-64798db975-sccdk\" (UID: \"85943ce7-bf4a-45e9-9342-36d6aae74b29\") " pod="openshift-ingress/kube-auth-proxy-64798db975-sccdk" Jul 08 18:51:29.571486 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:29.571453 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85943ce7-bf4a-45e9-9342-36d6aae74b29-tls-certs\") pod \"kube-auth-proxy-64798db975-sccdk\" (UID: \"85943ce7-bf4a-45e9-9342-36d6aae74b29\") " pod="openshift-ingress/kube-auth-proxy-64798db975-sccdk" Jul 08 18:51:29.583012 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:29.582990 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9xllh"] Jul 08 18:51:29.585310 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:51:29.585288 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf7227ff_4ae1_4f9c_8903_860139d8a70e.slice/crio-5b8dfa640b30f4b77df3f20450f085a7e22dcffb1a1b520479dfd60cda19cc7c WatchSource:0}: Error finding container 5b8dfa640b30f4b77df3f20450f085a7e22dcffb1a1b520479dfd60cda19cc7c: Status 404 returned error can't find the container with id 5b8dfa640b30f4b77df3f20450f085a7e22dcffb1a1b520479dfd60cda19cc7c Jul 08 18:51:29.784878 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:29.784775 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-64798db975-sccdk" Jul 08 18:51:29.920918 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:29.920893 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-64798db975-sccdk"] Jul 08 18:51:29.923216 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:51:29.923195 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85943ce7_bf4a_45e9_9342_36d6aae74b29.slice/crio-c56df3dea5bb72b8f3e04115d1471563c18bfef0a2fe00b05b8c2dade282275d WatchSource:0}: Error finding container c56df3dea5bb72b8f3e04115d1471563c18bfef0a2fe00b05b8c2dade282275d: Status 404 returned error can't find the container with id c56df3dea5bb72b8f3e04115d1471563c18bfef0a2fe00b05b8c2dade282275d Jul 08 18:51:30.208839 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:30.208753 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-9xllh" event={"ID":"df7227ff-4ae1-4f9c-8903-860139d8a70e","Type":"ContainerStarted","Data":"5b8dfa640b30f4b77df3f20450f085a7e22dcffb1a1b520479dfd60cda19cc7c"} Jul 08 18:51:30.210070 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:30.210043 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-64798db975-sccdk" event={"ID":"85943ce7-bf4a-45e9-9342-36d6aae74b29","Type":"ContainerStarted","Data":"c56df3dea5bb72b8f3e04115d1471563c18bfef0a2fe00b05b8c2dade282275d"} Jul 08 18:51:30.284097 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:30.284064 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s"] Jul 08 18:51:30.286782 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:30.286757 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s" Jul 08 18:51:30.290572 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:30.290548 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 08 18:51:30.290782 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:30.290751 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 08 18:51:30.290871 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:30.290790 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 08 18:51:30.290871 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:30.290795 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 08 18:51:30.290871 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:30.290846 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6r2pq\"" Jul 08 18:51:30.305876 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:30.305858 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s"] Jul 08 18:51:30.375537 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:30.375491 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcdcz\" (UniqueName: \"kubernetes.io/projected/7f2cba8c-046a-475d-87c4-e374810fda5f-kube-api-access-hcdcz\") pod \"istiod-openshift-gateway-7574459656-lvc4s\" (UID: \"7f2cba8c-046a-475d-87c4-e374810fda5f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s" Jul 08 18:51:30.375708 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:30.375563 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7f2cba8c-046a-475d-87c4-e374810fda5f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-lvc4s\" (UID: \"7f2cba8c-046a-475d-87c4-e374810fda5f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s" Jul 08 18:51:30.375708 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:30.375599 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f2cba8c-046a-475d-87c4-e374810fda5f-local-certs\") pod \"istiod-openshift-gateway-7574459656-lvc4s\" (UID: \"7f2cba8c-046a-475d-87c4-e374810fda5f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s" Jul 08 18:51:30.375708 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:30.375621 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7f2cba8c-046a-475d-87c4-e374810fda5f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-lvc4s\" (UID: \"7f2cba8c-046a-475d-87c4-e374810fda5f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s" Jul 08 18:51:30.375828 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:30.375732 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7f2cba8c-046a-475d-87c4-e374810fda5f-cacerts\") pod \"istiod-openshift-gateway-7574459656-lvc4s\" (UID: \"7f2cba8c-046a-475d-87c4-e374810fda5f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s" Jul 08 18:51:30.375828 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:30.375788 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f2cba8c-046a-475d-87c4-e374810fda5f-istio-token\") pod \"istiod-openshift-gateway-7574459656-lvc4s\" (UID: \"7f2cba8c-046a-475d-87c4-e374810fda5f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s" Jul 08 18:51:30.375828 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:30.375809 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7f2cba8c-046a-475d-87c4-e374810fda5f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-lvc4s\" (UID: \"7f2cba8c-046a-475d-87c4-e374810fda5f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s" Jul 08 18:51:30.476309 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:30.476204 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f2cba8c-046a-475d-87c4-e374810fda5f-istio-token\") pod \"istiod-openshift-gateway-7574459656-lvc4s\" (UID: \"7f2cba8c-046a-475d-87c4-e374810fda5f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s" Jul 08 18:51:30.476309 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:30.476283 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7f2cba8c-046a-475d-87c4-e374810fda5f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-lvc4s\" (UID: \"7f2cba8c-046a-475d-87c4-e374810fda5f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s" Jul 08 18:51:30.476528 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:30.476333 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcdcz\" (UniqueName: \"kubernetes.io/projected/7f2cba8c-046a-475d-87c4-e374810fda5f-kube-api-access-hcdcz\") pod \"istiod-openshift-gateway-7574459656-lvc4s\" (UID: \"7f2cba8c-046a-475d-87c4-e374810fda5f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s" Jul 08 18:51:30.476528 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:30.476385 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7f2cba8c-046a-475d-87c4-e374810fda5f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-lvc4s\" (UID: \"7f2cba8c-046a-475d-87c4-e374810fda5f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s" Jul 08 18:51:30.476528 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:30.476420 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f2cba8c-046a-475d-87c4-e374810fda5f-local-certs\") pod \"istiod-openshift-gateway-7574459656-lvc4s\" (UID: \"7f2cba8c-046a-475d-87c4-e374810fda5f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s" Jul 08 18:51:30.476528 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:30.476441 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7f2cba8c-046a-475d-87c4-e374810fda5f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-lvc4s\" (UID: \"7f2cba8c-046a-475d-87c4-e374810fda5f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s" Jul 08 18:51:30.476528 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:30.476517 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7f2cba8c-046a-475d-87c4-e374810fda5f-cacerts\") pod \"istiod-openshift-gateway-7574459656-lvc4s\" (UID: \"7f2cba8c-046a-475d-87c4-e374810fda5f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s" Jul 08 18:51:30.477487 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:30.477433 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7f2cba8c-046a-475d-87c4-e374810fda5f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-lvc4s\" (UID: \"7f2cba8c-046a-475d-87c4-e374810fda5f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s" Jul 08 18:51:30.481064 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:30.481013 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f2cba8c-046a-475d-87c4-e374810fda5f-local-certs\") pod \"istiod-openshift-gateway-7574459656-lvc4s\" (UID: \"7f2cba8c-046a-475d-87c4-e374810fda5f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s" Jul 08 18:51:30.481786 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:30.481755 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7f2cba8c-046a-475d-87c4-e374810fda5f-cacerts\") pod \"istiod-openshift-gateway-7574459656-lvc4s\" (UID: \"7f2cba8c-046a-475d-87c4-e374810fda5f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s" Jul 08 18:51:30.482560 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:30.482538 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7f2cba8c-046a-475d-87c4-e374810fda5f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-lvc4s\" (UID: \"7f2cba8c-046a-475d-87c4-e374810fda5f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s" Jul 08 18:51:30.482675 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:30.482567 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7f2cba8c-046a-475d-87c4-e374810fda5f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-lvc4s\" (UID: \"7f2cba8c-046a-475d-87c4-e374810fda5f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s" Jul 08 18:51:30.485945 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:30.485918 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcdcz\" (UniqueName: \"kubernetes.io/projected/7f2cba8c-046a-475d-87c4-e374810fda5f-kube-api-access-hcdcz\") pod \"istiod-openshift-gateway-7574459656-lvc4s\" (UID: \"7f2cba8c-046a-475d-87c4-e374810fda5f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s" Jul 08 18:51:30.486333 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:30.486315 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f2cba8c-046a-475d-87c4-e374810fda5f-istio-token\") pod \"istiod-openshift-gateway-7574459656-lvc4s\" (UID: \"7f2cba8c-046a-475d-87c4-e374810fda5f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s" Jul 08 18:51:30.598532 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:30.598489 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s" Jul 08 18:51:30.759950 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:30.759455 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s"] Jul 08 18:51:30.763216 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:51:30.763183 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f2cba8c_046a_475d_87c4_e374810fda5f.slice/crio-722b5f898ed8515c6137dbfb31343167a4e77ca4be14a77d29e76692809ea94a WatchSource:0}: Error finding container 722b5f898ed8515c6137dbfb31343167a4e77ca4be14a77d29e76692809ea94a: Status 404 returned error can't find the container with id 722b5f898ed8515c6137dbfb31343167a4e77ca4be14a77d29e76692809ea94a Jul 08 18:51:31.217319 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:31.217279 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s" event={"ID":"7f2cba8c-046a-475d-87c4-e374810fda5f","Type":"ContainerStarted","Data":"722b5f898ed8515c6137dbfb31343167a4e77ca4be14a77d29e76692809ea94a"} Jul 08 18:51:35.245670 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:35.245634 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 18:51:35.246036 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:35.245697 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 18:51:36.249218 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:36.249177 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s" event={"ID":"7f2cba8c-046a-475d-87c4-e374810fda5f","Type":"ContainerStarted","Data":"9e8df93f80557d911af74a87f37e9e417fcb42a1ba344adc1f3495cdbe3c5ecd"} Jul 08 18:51:36.249718 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:36.249243 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s" Jul 08 18:51:36.251064 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:36.251036 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-9xllh" event={"ID":"df7227ff-4ae1-4f9c-8903-860139d8a70e","Type":"ContainerStarted","Data":"67acd01a96028a92d2386f12f23b1c78a4f820781b0c4c06610eaf36130de960"} Jul 08 18:51:36.251171 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:36.251136 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-9xllh" Jul 08 18:51:36.251615 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:36.251596 2570 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-lvc4s container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 08 18:51:36.251733 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:36.251643 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s" podUID="7f2cba8c-046a-475d-87c4-e374810fda5f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 08 18:51:36.252595 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:36.252568 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-64798db975-sccdk" event={"ID":"85943ce7-bf4a-45e9-9342-36d6aae74b29","Type":"ContainerStarted","Data":"dd3b2652f395d4521a492a6a5125cf0a7f447b66f088232fe5599f4ed01e6429"} Jul 08 18:51:36.283734 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:36.283684 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s" podStartSLOduration=1.804228435 podStartE2EDuration="6.283670344s" totalImagesPullingTime="4.479441909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:51:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:51:36.282851957 +0000 UTC m=+357.130126622" watchObservedRunningTime="2026-07-08 18:51:36.283670344 +0000 UTC m=+357.130945004" Jul 08 18:51:36.312224 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:36.312177 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-64798db975-sccdk" podStartSLOduration=2.995187817 podStartE2EDuration="8.312163811s" totalImagesPullingTime="5.316975994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:51:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:51:36.30961561 +0000 UTC m=+357.156890276" watchObservedRunningTime="2026-07-08 18:51:36.312163811 +0000 UTC m=+357.159438477" Jul 08 18:51:36.354346 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:36.354289 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-9xllh" podStartSLOduration=1.702812449 podStartE2EDuration="7.354245539s" totalImagesPullingTime="5.65143309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:51:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:51:36.350848977 +0000 UTC m=+357.198123646" watchObservedRunningTime="2026-07-08 18:51:36.354245539 +0000 UTC m=+357.201520206" Jul 08 18:51:37.259668 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:37.259638 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s" Jul 08 18:51:41.436674 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:41.436638 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cffw4"] Jul 08 18:51:41.441868 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:41.441847 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cffw4" Jul 08 18:51:41.445820 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:41.445795 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-br7zk\"" Jul 08 18:51:41.447620 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:41.447603 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 08 18:51:41.455414 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:41.455393 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cffw4"] Jul 08 18:51:41.462505 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:41.462487 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-trp94"] Jul 08 18:51:41.465730 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:41.465713 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-trp94" Jul 08 18:51:41.469021 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:41.469002 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-brf5d\"" Jul 08 18:51:41.469192 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:41.469173 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 08 18:51:41.490333 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:41.490307 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-trp94"] Jul 08 18:51:41.579566 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:41.579532 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b7b3f4d-3e6f-4045-a0cb-041d055fe403-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-trp94\" (UID: \"0b7b3f4d-3e6f-4045-a0cb-041d055fe403\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-trp94" Jul 08 18:51:41.579740 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:41.579586 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b9ebce16-901d-48e0-90fd-42cb587f20fc-cert\") pod \"kserve-controller-manager-d6f4b65b8-cffw4\" (UID: \"b9ebce16-901d-48e0-90fd-42cb587f20fc\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cffw4" Jul 08 18:51:41.579740 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:41.579658 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hghc2\" (UniqueName: \"kubernetes.io/projected/b9ebce16-901d-48e0-90fd-42cb587f20fc-kube-api-access-hghc2\") pod \"kserve-controller-manager-d6f4b65b8-cffw4\" (UID: \"b9ebce16-901d-48e0-90fd-42cb587f20fc\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cffw4" Jul 08 18:51:41.579740 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:41.579712 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvfmv\" (UniqueName: \"kubernetes.io/projected/0b7b3f4d-3e6f-4045-a0cb-041d055fe403-kube-api-access-tvfmv\") pod \"llmisvc-controller-manager-777c6bd7d5-trp94\" (UID: \"0b7b3f4d-3e6f-4045-a0cb-041d055fe403\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-trp94" Jul 08 18:51:41.680231 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:41.680198 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hghc2\" (UniqueName: \"kubernetes.io/projected/b9ebce16-901d-48e0-90fd-42cb587f20fc-kube-api-access-hghc2\") pod \"kserve-controller-manager-d6f4b65b8-cffw4\" (UID: \"b9ebce16-901d-48e0-90fd-42cb587f20fc\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cffw4" Jul 08 18:51:41.680411 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:41.680268 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvfmv\" (UniqueName: \"kubernetes.io/projected/0b7b3f4d-3e6f-4045-a0cb-041d055fe403-kube-api-access-tvfmv\") pod \"llmisvc-controller-manager-777c6bd7d5-trp94\" (UID: \"0b7b3f4d-3e6f-4045-a0cb-041d055fe403\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-trp94" Jul 08 18:51:41.680411 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:41.680334 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b7b3f4d-3e6f-4045-a0cb-041d055fe403-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-trp94\" (UID: \"0b7b3f4d-3e6f-4045-a0cb-041d055fe403\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-trp94" Jul 08 18:51:41.680411 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:41.680372 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b9ebce16-901d-48e0-90fd-42cb587f20fc-cert\") pod \"kserve-controller-manager-d6f4b65b8-cffw4\" (UID: \"b9ebce16-901d-48e0-90fd-42cb587f20fc\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cffw4" Jul 08 18:51:41.680535 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:51:41.680479 2570 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 08 18:51:41.680572 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:51:41.680546 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b7b3f4d-3e6f-4045-a0cb-041d055fe403-cert podName:0b7b3f4d-3e6f-4045-a0cb-041d055fe403 nodeName:}" failed. No retries permitted until 2026-07-08 18:51:42.180529738 +0000 UTC m=+363.027804382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0b7b3f4d-3e6f-4045-a0cb-041d055fe403-cert") pod "llmisvc-controller-manager-777c6bd7d5-trp94" (UID: "0b7b3f4d-3e6f-4045-a0cb-041d055fe403") : secret "llmisvc-webhook-server-cert" not found Jul 08 18:51:41.682997 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:41.682971 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b9ebce16-901d-48e0-90fd-42cb587f20fc-cert\") pod \"kserve-controller-manager-d6f4b65b8-cffw4\" (UID: \"b9ebce16-901d-48e0-90fd-42cb587f20fc\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cffw4" Jul 08 18:51:41.694044 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:41.693979 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hghc2\" (UniqueName: \"kubernetes.io/projected/b9ebce16-901d-48e0-90fd-42cb587f20fc-kube-api-access-hghc2\") pod \"kserve-controller-manager-d6f4b65b8-cffw4\" (UID: \"b9ebce16-901d-48e0-90fd-42cb587f20fc\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cffw4" Jul 08 18:51:41.694153 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:41.694132 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvfmv\" (UniqueName: \"kubernetes.io/projected/0b7b3f4d-3e6f-4045-a0cb-041d055fe403-kube-api-access-tvfmv\") pod \"llmisvc-controller-manager-777c6bd7d5-trp94\" (UID: \"0b7b3f4d-3e6f-4045-a0cb-041d055fe403\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-trp94" Jul 08 18:51:41.753156 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:41.753121 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cffw4" Jul 08 18:51:41.898533 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:41.898511 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cffw4"] Jul 08 18:51:41.901761 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:51:41.901734 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9ebce16_901d_48e0_90fd_42cb587f20fc.slice/crio-b7001b2a070ced5930d3a83984ccda6ae31dba0c60321155b2fe90c3e7cdc54b WatchSource:0}: Error finding container b7001b2a070ced5930d3a83984ccda6ae31dba0c60321155b2fe90c3e7cdc54b: Status 404 returned error can't find the container with id b7001b2a070ced5930d3a83984ccda6ae31dba0c60321155b2fe90c3e7cdc54b Jul 08 18:51:42.183778 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:42.183737 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b7b3f4d-3e6f-4045-a0cb-041d055fe403-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-trp94\" (UID: \"0b7b3f4d-3e6f-4045-a0cb-041d055fe403\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-trp94" Jul 08 18:51:42.186029 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:42.186006 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b7b3f4d-3e6f-4045-a0cb-041d055fe403-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-trp94\" (UID: \"0b7b3f4d-3e6f-4045-a0cb-041d055fe403\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-trp94" Jul 08 18:51:42.288494 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:42.288460 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cffw4" event={"ID":"b9ebce16-901d-48e0-90fd-42cb587f20fc","Type":"ContainerStarted","Data":"b7001b2a070ced5930d3a83984ccda6ae31dba0c60321155b2fe90c3e7cdc54b"} Jul 08 18:51:42.375491 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:42.375460 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-trp94" Jul 08 18:51:42.518889 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:42.518864 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-trp94"] Jul 08 18:51:42.521055 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:51:42.521027 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0b7b3f4d_3e6f_4045_a0cb_041d055fe403.slice/crio-0c144f8841e8d3ab8d551073a4bf657bfadab6b7cd04c64b49af1c00af3598cf WatchSource:0}: Error finding container 0c144f8841e8d3ab8d551073a4bf657bfadab6b7cd04c64b49af1c00af3598cf: Status 404 returned error can't find the container with id 0c144f8841e8d3ab8d551073a4bf657bfadab6b7cd04c64b49af1c00af3598cf Jul 08 18:51:42.538356 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:42.538334 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s"] Jul 08 18:51:42.538525 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:42.538490 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s" podUID="7f2cba8c-046a-475d-87c4-e374810fda5f" containerName="discovery" containerID="cri-o://9e8df93f80557d911af74a87f37e9e417fcb42a1ba344adc1f3495cdbe3c5ecd" gracePeriod=30 Jul 08 18:51:42.812147 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:42.812122 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s" Jul 08 18:51:42.890653 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:42.890625 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7f2cba8c-046a-475d-87c4-e374810fda5f-cacerts\") pod \"7f2cba8c-046a-475d-87c4-e374810fda5f\" (UID: \"7f2cba8c-046a-475d-87c4-e374810fda5f\") " Jul 08 18:51:42.890817 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:42.890681 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f2cba8c-046a-475d-87c4-e374810fda5f-istio-token\") pod \"7f2cba8c-046a-475d-87c4-e374810fda5f\" (UID: \"7f2cba8c-046a-475d-87c4-e374810fda5f\") " Jul 08 18:51:42.890817 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:42.890706 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7f2cba8c-046a-475d-87c4-e374810fda5f-istio-kubeconfig\") pod \"7f2cba8c-046a-475d-87c4-e374810fda5f\" (UID: \"7f2cba8c-046a-475d-87c4-e374810fda5f\") " Jul 08 18:51:42.890817 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:42.890781 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcdcz\" (UniqueName: \"kubernetes.io/projected/7f2cba8c-046a-475d-87c4-e374810fda5f-kube-api-access-hcdcz\") pod \"7f2cba8c-046a-475d-87c4-e374810fda5f\" (UID: \"7f2cba8c-046a-475d-87c4-e374810fda5f\") " Jul 08 18:51:42.890817 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:42.890805 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7f2cba8c-046a-475d-87c4-e374810fda5f-istio-csr-ca-configmap\") pod \"7f2cba8c-046a-475d-87c4-e374810fda5f\" (UID: \"7f2cba8c-046a-475d-87c4-e374810fda5f\") " Jul 08 18:51:42.891021 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:42.890841 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f2cba8c-046a-475d-87c4-e374810fda5f-local-certs\") pod \"7f2cba8c-046a-475d-87c4-e374810fda5f\" (UID: \"7f2cba8c-046a-475d-87c4-e374810fda5f\") " Jul 08 18:51:42.891021 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:42.890943 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7f2cba8c-046a-475d-87c4-e374810fda5f-istio-csr-dns-cert\") pod \"7f2cba8c-046a-475d-87c4-e374810fda5f\" (UID: \"7f2cba8c-046a-475d-87c4-e374810fda5f\") " Jul 08 18:51:42.891304 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:42.891245 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f2cba8c-046a-475d-87c4-e374810fda5f-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "7f2cba8c-046a-475d-87c4-e374810fda5f" (UID: "7f2cba8c-046a-475d-87c4-e374810fda5f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:51:42.893017 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:42.892994 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f2cba8c-046a-475d-87c4-e374810fda5f-istio-token" (OuterVolumeSpecName: "istio-token") pod "7f2cba8c-046a-475d-87c4-e374810fda5f" (UID: "7f2cba8c-046a-475d-87c4-e374810fda5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:51:42.893181 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:42.893166 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f2cba8c-046a-475d-87c4-e374810fda5f-cacerts" (OuterVolumeSpecName: "cacerts") pod "7f2cba8c-046a-475d-87c4-e374810fda5f" (UID: "7f2cba8c-046a-475d-87c4-e374810fda5f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:51:42.893238 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:42.893218 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2cba8c-046a-475d-87c4-e374810fda5f-local-certs" (OuterVolumeSpecName: "local-certs") pod "7f2cba8c-046a-475d-87c4-e374810fda5f" (UID: "7f2cba8c-046a-475d-87c4-e374810fda5f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:51:42.893389 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:42.893368 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f2cba8c-046a-475d-87c4-e374810fda5f-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "7f2cba8c-046a-475d-87c4-e374810fda5f" (UID: "7f2cba8c-046a-475d-87c4-e374810fda5f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:51:42.893494 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:42.893477 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f2cba8c-046a-475d-87c4-e374810fda5f-kube-api-access-hcdcz" (OuterVolumeSpecName: "kube-api-access-hcdcz") pod "7f2cba8c-046a-475d-87c4-e374810fda5f" (UID: "7f2cba8c-046a-475d-87c4-e374810fda5f"). InnerVolumeSpecName "kube-api-access-hcdcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:51:42.893598 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:42.893585 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f2cba8c-046a-475d-87c4-e374810fda5f-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "7f2cba8c-046a-475d-87c4-e374810fda5f" (UID: "7f2cba8c-046a-475d-87c4-e374810fda5f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:51:42.992482 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:42.992407 2570 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f2cba8c-046a-475d-87c4-e374810fda5f-local-certs\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:51:42.992482 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:42.992442 2570 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7f2cba8c-046a-475d-87c4-e374810fda5f-istio-csr-dns-cert\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:51:42.992482 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:42.992451 2570 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7f2cba8c-046a-475d-87c4-e374810fda5f-cacerts\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:51:42.992482 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:42.992459 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f2cba8c-046a-475d-87c4-e374810fda5f-istio-token\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:51:42.992482 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:42.992468 2570 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7f2cba8c-046a-475d-87c4-e374810fda5f-istio-kubeconfig\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:51:42.992482 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:42.992476 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcdcz\" (UniqueName: \"kubernetes.io/projected/7f2cba8c-046a-475d-87c4-e374810fda5f-kube-api-access-hcdcz\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:51:42.992482 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:42.992484 2570 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7f2cba8c-046a-475d-87c4-e374810fda5f-istio-csr-ca-configmap\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:51:43.294564 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:43.294495 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-trp94" event={"ID":"0b7b3f4d-3e6f-4045-a0cb-041d055fe403","Type":"ContainerStarted","Data":"0c144f8841e8d3ab8d551073a4bf657bfadab6b7cd04c64b49af1c00af3598cf"} Jul 08 18:51:43.295710 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:43.295683 2570 generic.go:356] "Generic (PLEG): container finished" podID="7f2cba8c-046a-475d-87c4-e374810fda5f" containerID="9e8df93f80557d911af74a87f37e9e417fcb42a1ba344adc1f3495cdbe3c5ecd" exitCode=0 Jul 08 18:51:43.295848 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:43.295721 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s" Jul 08 18:51:43.295848 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:43.295764 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s" event={"ID":"7f2cba8c-046a-475d-87c4-e374810fda5f","Type":"ContainerDied","Data":"9e8df93f80557d911af74a87f37e9e417fcb42a1ba344adc1f3495cdbe3c5ecd"} Jul 08 18:51:43.295848 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:43.295799 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s" event={"ID":"7f2cba8c-046a-475d-87c4-e374810fda5f","Type":"ContainerDied","Data":"722b5f898ed8515c6137dbfb31343167a4e77ca4be14a77d29e76692809ea94a"} Jul 08 18:51:43.295848 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:43.295818 2570 scope.go:117] "RemoveContainer" containerID="9e8df93f80557d911af74a87f37e9e417fcb42a1ba344adc1f3495cdbe3c5ecd" Jul 08 18:51:43.305115 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:43.305097 2570 scope.go:117] "RemoveContainer" containerID="9e8df93f80557d911af74a87f37e9e417fcb42a1ba344adc1f3495cdbe3c5ecd" Jul 08 18:51:43.305396 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:51:43.305378 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e8df93f80557d911af74a87f37e9e417fcb42a1ba344adc1f3495cdbe3c5ecd\": container with ID starting with 9e8df93f80557d911af74a87f37e9e417fcb42a1ba344adc1f3495cdbe3c5ecd not found: ID does not exist" containerID="9e8df93f80557d911af74a87f37e9e417fcb42a1ba344adc1f3495cdbe3c5ecd" Jul 08 18:51:43.305479 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:43.305404 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e8df93f80557d911af74a87f37e9e417fcb42a1ba344adc1f3495cdbe3c5ecd"} err="failed to get container status \"9e8df93f80557d911af74a87f37e9e417fcb42a1ba344adc1f3495cdbe3c5ecd\": rpc error: code = NotFound desc = could not find container \"9e8df93f80557d911af74a87f37e9e417fcb42a1ba344adc1f3495cdbe3c5ecd\": container with ID starting with 9e8df93f80557d911af74a87f37e9e417fcb42a1ba344adc1f3495cdbe3c5ecd not found: ID does not exist" Jul 08 18:51:43.332413 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:43.332385 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s"] Jul 08 18:51:43.343707 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:43.343685 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-lvc4s"] Jul 08 18:51:43.691299 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:43.691242 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f2cba8c-046a-475d-87c4-e374810fda5f" path="/var/lib/kubelet/pods/7f2cba8c-046a-475d-87c4-e374810fda5f/volumes" Jul 08 18:51:45.317479 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:45.317383 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cffw4" event={"ID":"b9ebce16-901d-48e0-90fd-42cb587f20fc","Type":"ContainerStarted","Data":"dbee6b12256ddb4a491345622e438176770ac4b6f8ca0adbb2d2310c9d1d3b7a"} Jul 08 18:51:45.317918 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:45.317525 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cffw4" Jul 08 18:51:45.340934 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:45.340872 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cffw4" podStartSLOduration=1.202137322 podStartE2EDuration="4.340852734s" totalImagesPullingTime="3.138715412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:51:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:51:45.33904394 +0000 UTC m=+366.186318601" watchObservedRunningTime="2026-07-08 18:51:45.340852734 +0000 UTC m=+366.188127401" Jul 08 18:51:46.324208 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:46.324122 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-trp94" event={"ID":"0b7b3f4d-3e6f-4045-a0cb-041d055fe403","Type":"ContainerStarted","Data":"52ac8ef12649fbc75283fa66eb1873cb2be3f9e80657bf5989d01d640dab3cfa"} Jul 08 18:51:46.343412 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:46.343367 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-trp94" podStartSLOduration=1.906332249 podStartE2EDuration="5.343352885s" totalImagesPullingTime="3.437020636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:51:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:51:46.341625678 +0000 UTC m=+367.188900346" watchObservedRunningTime="2026-07-08 18:51:46.343352885 +0000 UTC m=+367.190627552" Jul 08 18:51:47.260480 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:47.260451 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-9xllh" Jul 08 18:51:47.328779 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:51:47.328750 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-trp94" Jul 08 18:52:16.330343 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:16.330312 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cffw4" Jul 08 18:52:18.338157 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:18.338126 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-trp94" Jul 08 18:52:29.717033 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:29.716978 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xkgmv"] Jul 08 18:52:29.717537 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:29.717391 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f2cba8c-046a-475d-87c4-e374810fda5f" containerName="discovery" Jul 08 18:52:29.717537 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:29.717405 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f2cba8c-046a-475d-87c4-e374810fda5f" containerName="discovery" Jul 08 18:52:29.717537 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:29.717489 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f2cba8c-046a-475d-87c4-e374810fda5f" containerName="discovery" Jul 08 18:52:29.726025 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:29.725996 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xkgmv" Jul 08 18:52:29.728703 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:29.728678 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 08 18:52:29.728834 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:29.728701 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 08 18:52:29.729498 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:29.729479 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-vqd8r\"" Jul 08 18:52:29.735297 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:29.735275 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xkgmv"] Jul 08 18:52:29.785036 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:29.784839 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84b6z\" (UniqueName: \"kubernetes.io/projected/cb22df44-d246-4816-accc-cf9086a85efc-kube-api-access-84b6z\") pod \"kuadrant-operator-catalog-xkgmv\" (UID: \"cb22df44-d246-4816-accc-cf9086a85efc\") " pod="kuadrant-system/kuadrant-operator-catalog-xkgmv" Jul 08 18:52:29.886367 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:29.886332 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84b6z\" (UniqueName: \"kubernetes.io/projected/cb22df44-d246-4816-accc-cf9086a85efc-kube-api-access-84b6z\") pod \"kuadrant-operator-catalog-xkgmv\" (UID: \"cb22df44-d246-4816-accc-cf9086a85efc\") " pod="kuadrant-system/kuadrant-operator-catalog-xkgmv" Jul 08 18:52:29.895566 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:29.895530 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84b6z\" (UniqueName: \"kubernetes.io/projected/cb22df44-d246-4816-accc-cf9086a85efc-kube-api-access-84b6z\") pod \"kuadrant-operator-catalog-xkgmv\" (UID: \"cb22df44-d246-4816-accc-cf9086a85efc\") " pod="kuadrant-system/kuadrant-operator-catalog-xkgmv" Jul 08 18:52:30.037947 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:30.037846 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xkgmv" Jul 08 18:52:30.076053 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:30.076013 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xkgmv"] Jul 08 18:52:30.197071 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:30.196930 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xkgmv"] Jul 08 18:52:30.199963 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:52:30.199931 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb22df44_d246_4816_accc_cf9086a85efc.slice/crio-7f27e240f76cdeecf630078025fb33a56f943c003d72f021eb8c29f97b438fad WatchSource:0}: Error finding container 7f27e240f76cdeecf630078025fb33a56f943c003d72f021eb8c29f97b438fad: Status 404 returned error can't find the container with id 7f27e240f76cdeecf630078025fb33a56f943c003d72f021eb8c29f97b438fad Jul 08 18:52:30.286051 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:30.286011 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2q2kh"] Jul 08 18:52:30.288913 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:30.288853 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2q2kh" Jul 08 18:52:30.298046 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:30.298021 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2q2kh"] Jul 08 18:52:30.391299 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:30.391238 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ccr5\" (UniqueName: \"kubernetes.io/projected/7c02bba5-df73-4c69-ac59-269c874ecf10-kube-api-access-4ccr5\") pod \"kuadrant-operator-catalog-2q2kh\" (UID: \"7c02bba5-df73-4c69-ac59-269c874ecf10\") " pod="kuadrant-system/kuadrant-operator-catalog-2q2kh" Jul 08 18:52:30.491877 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:30.491838 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ccr5\" (UniqueName: \"kubernetes.io/projected/7c02bba5-df73-4c69-ac59-269c874ecf10-kube-api-access-4ccr5\") pod \"kuadrant-operator-catalog-2q2kh\" (UID: \"7c02bba5-df73-4c69-ac59-269c874ecf10\") " pod="kuadrant-system/kuadrant-operator-catalog-2q2kh" Jul 08 18:52:30.502177 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:30.502146 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ccr5\" (UniqueName: \"kubernetes.io/projected/7c02bba5-df73-4c69-ac59-269c874ecf10-kube-api-access-4ccr5\") pod \"kuadrant-operator-catalog-2q2kh\" (UID: \"7c02bba5-df73-4c69-ac59-269c874ecf10\") " pod="kuadrant-system/kuadrant-operator-catalog-2q2kh" Jul 08 18:52:30.527284 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:30.527217 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xkgmv" event={"ID":"cb22df44-d246-4816-accc-cf9086a85efc","Type":"ContainerStarted","Data":"7f27e240f76cdeecf630078025fb33a56f943c003d72f021eb8c29f97b438fad"} Jul 08 18:52:30.599733 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:30.599643 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2q2kh" Jul 08 18:52:30.795064 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:30.795030 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2q2kh"] Jul 08 18:52:30.822279 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:52:30.822232 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c02bba5_df73_4c69_ac59_269c874ecf10.slice/crio-02e0033218ae31736378569b711898950395f801fd1d344cc3f0a4bfbce389a5 WatchSource:0}: Error finding container 02e0033218ae31736378569b711898950395f801fd1d344cc3f0a4bfbce389a5: Status 404 returned error can't find the container with id 02e0033218ae31736378569b711898950395f801fd1d344cc3f0a4bfbce389a5 Jul 08 18:52:31.534420 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:31.534377 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2q2kh" event={"ID":"7c02bba5-df73-4c69-ac59-269c874ecf10","Type":"ContainerStarted","Data":"02e0033218ae31736378569b711898950395f801fd1d344cc3f0a4bfbce389a5"} Jul 08 18:52:32.541045 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:32.540936 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xkgmv" event={"ID":"cb22df44-d246-4816-accc-cf9086a85efc","Type":"ContainerStarted","Data":"16b3e50647ddd6dba7336bafad0ee6ba2981ac1dedc5141bdfc7090dd5950441"} Jul 08 18:52:32.541045 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:32.540952 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-xkgmv" podUID="cb22df44-d246-4816-accc-cf9086a85efc" containerName="registry-server" containerID="cri-o://16b3e50647ddd6dba7336bafad0ee6ba2981ac1dedc5141bdfc7090dd5950441" gracePeriod=2 Jul 08 18:52:32.542577 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:32.542552 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2q2kh" event={"ID":"7c02bba5-df73-4c69-ac59-269c874ecf10","Type":"ContainerStarted","Data":"6d3443db746f29b6726f09b8d3fc30753b879db50bfc187b6ddace3c48dfbf52"} Jul 08 18:52:32.559059 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:32.559014 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-xkgmv" podStartSLOduration=1.619900061 podStartE2EDuration="3.558998748s" totalImagesPullingTime="1.939098687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:52:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:52:32.557612136 +0000 UTC m=+413.404886796" watchObservedRunningTime="2026-07-08 18:52:32.558998748 +0000 UTC m=+413.406273414" Jul 08 18:52:32.577031 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:32.576974 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-2q2kh" podStartSLOduration=1.257858215 podStartE2EDuration="2.576957676s" totalImagesPullingTime="1.319099461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:52:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:52:32.575306545 +0000 UTC m=+413.422581214" watchObservedRunningTime="2026-07-08 18:52:32.576957676 +0000 UTC m=+413.424232342" Jul 08 18:52:32.790363 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:32.790338 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xkgmv" Jul 08 18:52:32.914027 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:32.913992 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84b6z\" (UniqueName: \"kubernetes.io/projected/cb22df44-d246-4816-accc-cf9086a85efc-kube-api-access-84b6z\") pod \"cb22df44-d246-4816-accc-cf9086a85efc\" (UID: \"cb22df44-d246-4816-accc-cf9086a85efc\") " Jul 08 18:52:32.916219 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:32.916189 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb22df44-d246-4816-accc-cf9086a85efc-kube-api-access-84b6z" (OuterVolumeSpecName: "kube-api-access-84b6z") pod "cb22df44-d246-4816-accc-cf9086a85efc" (UID: "cb22df44-d246-4816-accc-cf9086a85efc"). InnerVolumeSpecName "kube-api-access-84b6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:52:33.015238 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:33.015188 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84b6z\" (UniqueName: \"kubernetes.io/projected/cb22df44-d246-4816-accc-cf9086a85efc-kube-api-access-84b6z\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:52:33.548942 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:33.548856 2570 generic.go:356] "Generic (PLEG): container finished" podID="cb22df44-d246-4816-accc-cf9086a85efc" containerID="16b3e50647ddd6dba7336bafad0ee6ba2981ac1dedc5141bdfc7090dd5950441" exitCode=0 Jul 08 18:52:33.548942 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:33.548897 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xkgmv" Jul 08 18:52:33.548942 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:33.548932 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xkgmv" event={"ID":"cb22df44-d246-4816-accc-cf9086a85efc","Type":"ContainerDied","Data":"16b3e50647ddd6dba7336bafad0ee6ba2981ac1dedc5141bdfc7090dd5950441"} Jul 08 18:52:33.549525 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:33.548980 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xkgmv" event={"ID":"cb22df44-d246-4816-accc-cf9086a85efc","Type":"ContainerDied","Data":"7f27e240f76cdeecf630078025fb33a56f943c003d72f021eb8c29f97b438fad"} Jul 08 18:52:33.549525 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:33.549010 2570 scope.go:117] "RemoveContainer" containerID="16b3e50647ddd6dba7336bafad0ee6ba2981ac1dedc5141bdfc7090dd5950441" Jul 08 18:52:33.558338 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:33.558321 2570 scope.go:117] "RemoveContainer" containerID="16b3e50647ddd6dba7336bafad0ee6ba2981ac1dedc5141bdfc7090dd5950441" Jul 08 18:52:33.558583 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:52:33.558564 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16b3e50647ddd6dba7336bafad0ee6ba2981ac1dedc5141bdfc7090dd5950441\": container with ID starting with 16b3e50647ddd6dba7336bafad0ee6ba2981ac1dedc5141bdfc7090dd5950441 not found: ID does not exist" containerID="16b3e50647ddd6dba7336bafad0ee6ba2981ac1dedc5141bdfc7090dd5950441" Jul 08 18:52:33.558628 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:33.558591 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16b3e50647ddd6dba7336bafad0ee6ba2981ac1dedc5141bdfc7090dd5950441"} err="failed to get container status \"16b3e50647ddd6dba7336bafad0ee6ba2981ac1dedc5141bdfc7090dd5950441\": rpc error: code = NotFound desc = could not find container \"16b3e50647ddd6dba7336bafad0ee6ba2981ac1dedc5141bdfc7090dd5950441\": container with ID starting with 16b3e50647ddd6dba7336bafad0ee6ba2981ac1dedc5141bdfc7090dd5950441 not found: ID does not exist" Jul 08 18:52:33.573601 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:33.571506 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xkgmv"] Jul 08 18:52:33.577439 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:33.577419 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xkgmv"] Jul 08 18:52:33.689335 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:33.689300 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb22df44-d246-4816-accc-cf9086a85efc" path="/var/lib/kubelet/pods/cb22df44-d246-4816-accc-cf9086a85efc/volumes" Jul 08 18:52:37.386025 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:37.385964 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68d45ff57c-7m45k"] Jul 08 18:52:37.386463 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:37.386437 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb22df44-d246-4816-accc-cf9086a85efc" containerName="registry-server" Jul 08 18:52:37.386463 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:37.386452 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb22df44-d246-4816-accc-cf9086a85efc" containerName="registry-server" Jul 08 18:52:37.386535 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:37.386506 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb22df44-d246-4816-accc-cf9086a85efc" containerName="registry-server" Jul 08 18:52:37.390894 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:37.390876 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68d45ff57c-7m45k" Jul 08 18:52:37.393565 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:37.393543 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 08 18:52:37.393668 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:37.393544 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 08 18:52:37.394457 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:37.394431 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 08 18:52:37.394591 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:37.394479 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wqtlv\"" Jul 08 18:52:37.394665 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:37.394602 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 08 18:52:37.394779 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:37.394762 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 08 18:52:37.398973 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:37.398951 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 08 18:52:37.402588 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:37.402570 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68d45ff57c-7m45k"] Jul 08 18:52:37.555119 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:37.555090 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzbg6\" (UniqueName: \"kubernetes.io/projected/46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa-kube-api-access-hzbg6\") pod \"console-68d45ff57c-7m45k\" (UID: \"46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa\") " pod="openshift-console/console-68d45ff57c-7m45k" Jul 08 18:52:37.555330 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:37.555150 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa-oauth-serving-cert\") pod \"console-68d45ff57c-7m45k\" (UID: \"46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa\") " pod="openshift-console/console-68d45ff57c-7m45k" Jul 08 18:52:37.555330 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:37.555210 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa-console-oauth-config\") pod \"console-68d45ff57c-7m45k\" (UID: \"46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa\") " pod="openshift-console/console-68d45ff57c-7m45k" Jul 08 18:52:37.555330 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:37.555243 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa-trusted-ca-bundle\") pod \"console-68d45ff57c-7m45k\" (UID: \"46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa\") " pod="openshift-console/console-68d45ff57c-7m45k" Jul 08 18:52:37.555330 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:37.555297 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa-console-config\") pod \"console-68d45ff57c-7m45k\" (UID: \"46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa\") " pod="openshift-console/console-68d45ff57c-7m45k" Jul 08 18:52:37.555482 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:37.555335 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa-console-serving-cert\") pod \"console-68d45ff57c-7m45k\" (UID: \"46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa\") " pod="openshift-console/console-68d45ff57c-7m45k" Jul 08 18:52:37.555482 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:37.555369 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa-service-ca\") pod \"console-68d45ff57c-7m45k\" (UID: \"46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa\") " pod="openshift-console/console-68d45ff57c-7m45k" Jul 08 18:52:37.655851 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:37.655756 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzbg6\" (UniqueName: \"kubernetes.io/projected/46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa-kube-api-access-hzbg6\") pod \"console-68d45ff57c-7m45k\" (UID: \"46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa\") " pod="openshift-console/console-68d45ff57c-7m45k" Jul 08 18:52:37.655851 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:37.655804 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa-oauth-serving-cert\") pod \"console-68d45ff57c-7m45k\" (UID: \"46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa\") " pod="openshift-console/console-68d45ff57c-7m45k" Jul 08 18:52:37.655851 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:37.655835 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa-console-oauth-config\") pod \"console-68d45ff57c-7m45k\" (UID: \"46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa\") " pod="openshift-console/console-68d45ff57c-7m45k" Jul 08 18:52:37.655851 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:37.655852 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa-trusted-ca-bundle\") pod \"console-68d45ff57c-7m45k\" (UID: \"46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa\") " pod="openshift-console/console-68d45ff57c-7m45k" Jul 08 18:52:37.656184 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:37.655878 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa-console-config\") pod \"console-68d45ff57c-7m45k\" (UID: \"46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa\") " pod="openshift-console/console-68d45ff57c-7m45k" Jul 08 18:52:37.656184 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:37.655896 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa-console-serving-cert\") pod \"console-68d45ff57c-7m45k\" (UID: \"46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa\") " pod="openshift-console/console-68d45ff57c-7m45k" Jul 08 18:52:37.656184 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:37.655931 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa-service-ca\") pod \"console-68d45ff57c-7m45k\" (UID: \"46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa\") " pod="openshift-console/console-68d45ff57c-7m45k" Jul 08 18:52:37.656628 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:37.656602 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa-oauth-serving-cert\") pod \"console-68d45ff57c-7m45k\" (UID: \"46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa\") " pod="openshift-console/console-68d45ff57c-7m45k" Jul 08 18:52:37.656746 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:37.656700 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa-console-config\") pod \"console-68d45ff57c-7m45k\" (UID: \"46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa\") " pod="openshift-console/console-68d45ff57c-7m45k" Jul 08 18:52:37.656746 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:37.656702 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa-service-ca\") pod \"console-68d45ff57c-7m45k\" (UID: \"46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa\") " pod="openshift-console/console-68d45ff57c-7m45k" Jul 08 18:52:37.656832 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:37.656817 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa-trusted-ca-bundle\") pod \"console-68d45ff57c-7m45k\" (UID: \"46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa\") " pod="openshift-console/console-68d45ff57c-7m45k" Jul 08 18:52:37.658870 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:37.658847 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa-console-oauth-config\") pod \"console-68d45ff57c-7m45k\" (UID: \"46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa\") " pod="openshift-console/console-68d45ff57c-7m45k" Jul 08 18:52:37.659067 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:37.659047 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa-console-serving-cert\") pod \"console-68d45ff57c-7m45k\" (UID: \"46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa\") " pod="openshift-console/console-68d45ff57c-7m45k" Jul 08 18:52:37.665440 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:37.665417 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzbg6\" (UniqueName: \"kubernetes.io/projected/46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa-kube-api-access-hzbg6\") pod \"console-68d45ff57c-7m45k\" (UID: \"46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa\") " pod="openshift-console/console-68d45ff57c-7m45k" Jul 08 18:52:37.699679 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:37.699657 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68d45ff57c-7m45k" Jul 08 18:52:37.832460 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:37.832431 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68d45ff57c-7m45k"] Jul 08 18:52:37.835068 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:52:37.835029 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46ebf7d1_1bc3_43de_bfd1_a4d7d03babfa.slice/crio-7cae26b3669dce9faccc096f6a9075712af9ff7677420d4b4181b626d2211429 WatchSource:0}: Error finding container 7cae26b3669dce9faccc096f6a9075712af9ff7677420d4b4181b626d2211429: Status 404 returned error can't find the container with id 7cae26b3669dce9faccc096f6a9075712af9ff7677420d4b4181b626d2211429 Jul 08 18:52:38.574988 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:38.574946 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68d45ff57c-7m45k" event={"ID":"46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa","Type":"ContainerStarted","Data":"5912ae9cda6e2f27ee4ddc6654c9e8ef264318d100811240896d89f9001ed678"} Jul 08 18:52:38.575463 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:38.575043 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68d45ff57c-7m45k" event={"ID":"46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa","Type":"ContainerStarted","Data":"7cae26b3669dce9faccc096f6a9075712af9ff7677420d4b4181b626d2211429"} Jul 08 18:52:38.602935 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:38.602883 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68d45ff57c-7m45k" podStartSLOduration=1.6028690700000001 podStartE2EDuration="1.60286907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:52:38.602691729 +0000 UTC m=+419.449966395" watchObservedRunningTime="2026-07-08 18:52:38.60286907 +0000 UTC m=+419.450143736" Jul 08 18:52:40.600864 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:40.600816 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-2q2kh" Jul 08 18:52:40.601351 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:40.600893 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-2q2kh" Jul 08 18:52:40.623782 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:40.623746 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-2q2kh" Jul 08 18:52:41.610889 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:41.610855 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-2q2kh" Jul 08 18:52:47.699828 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:47.699791 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68d45ff57c-7m45k" Jul 08 18:52:47.700372 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:47.699848 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68d45ff57c-7m45k" Jul 08 18:52:47.704512 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:47.704485 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68d45ff57c-7m45k" Jul 08 18:52:48.627633 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:48.627605 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68d45ff57c-7m45k" Jul 08 18:52:59.472777 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:59.472742 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vbp48"] Jul 08 18:52:59.477577 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:59.477560 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vbp48" Jul 08 18:52:59.481443 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:59.481422 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8c9hs\"" Jul 08 18:52:59.491911 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:59.491891 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vbp48"] Jul 08 18:52:59.537057 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:59.537017 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/55c0a620-41c0-4f36-ac2d-8a7f649b0216-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vbp48\" (UID: \"55c0a620-41c0-4f36-ac2d-8a7f649b0216\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vbp48" Jul 08 18:52:59.537213 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:59.537081 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/55c0a620-41c0-4f36-ac2d-8a7f649b0216-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vbp48\" (UID: \"55c0a620-41c0-4f36-ac2d-8a7f649b0216\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vbp48" Jul 08 18:52:59.537213 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:59.537155 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwq5r\" (UniqueName: \"kubernetes.io/projected/55c0a620-41c0-4f36-ac2d-8a7f649b0216-kube-api-access-zwq5r\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vbp48\" (UID: \"55c0a620-41c0-4f36-ac2d-8a7f649b0216\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vbp48" Jul 08 18:52:59.595359 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:59.595325 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bq554"] Jul 08 18:52:59.598865 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:59.598849 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bq554" Jul 08 18:52:59.620013 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:59.619977 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bq554"] Jul 08 18:52:59.637550 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:59.637526 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c2e7ae4-1be9-450b-9d0b-52d16d2108b8-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bq554\" (UID: \"3c2e7ae4-1be9-450b-9d0b-52d16d2108b8\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bq554" Jul 08 18:52:59.637705 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:59.637563 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwq5r\" (UniqueName: \"kubernetes.io/projected/55c0a620-41c0-4f36-ac2d-8a7f649b0216-kube-api-access-zwq5r\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vbp48\" (UID: \"55c0a620-41c0-4f36-ac2d-8a7f649b0216\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vbp48" Jul 08 18:52:59.637705 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:59.637598 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c2e7ae4-1be9-450b-9d0b-52d16d2108b8-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bq554\" (UID: \"3c2e7ae4-1be9-450b-9d0b-52d16d2108b8\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bq554" Jul 08 18:52:59.637705 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:59.637623 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzstq\" (UniqueName: \"kubernetes.io/projected/3c2e7ae4-1be9-450b-9d0b-52d16d2108b8-kube-api-access-wzstq\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bq554\" (UID: \"3c2e7ae4-1be9-450b-9d0b-52d16d2108b8\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bq554" Jul 08 18:52:59.637705 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:59.637663 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/55c0a620-41c0-4f36-ac2d-8a7f649b0216-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vbp48\" (UID: \"55c0a620-41c0-4f36-ac2d-8a7f649b0216\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vbp48" Jul 08 18:52:59.637886 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:59.637753 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/55c0a620-41c0-4f36-ac2d-8a7f649b0216-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vbp48\" (UID: \"55c0a620-41c0-4f36-ac2d-8a7f649b0216\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vbp48" Jul 08 18:52:59.638079 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:59.638042 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/55c0a620-41c0-4f36-ac2d-8a7f649b0216-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vbp48\" (UID: \"55c0a620-41c0-4f36-ac2d-8a7f649b0216\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vbp48" Jul 08 18:52:59.638079 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:59.638067 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/55c0a620-41c0-4f36-ac2d-8a7f649b0216-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vbp48\" (UID: \"55c0a620-41c0-4f36-ac2d-8a7f649b0216\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vbp48" Jul 08 18:52:59.647106 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:59.647082 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwq5r\" (UniqueName: \"kubernetes.io/projected/55c0a620-41c0-4f36-ac2d-8a7f649b0216-kube-api-access-zwq5r\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vbp48\" (UID: \"55c0a620-41c0-4f36-ac2d-8a7f649b0216\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vbp48" Jul 08 18:52:59.739183 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:59.739097 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c2e7ae4-1be9-450b-9d0b-52d16d2108b8-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bq554\" (UID: \"3c2e7ae4-1be9-450b-9d0b-52d16d2108b8\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bq554" Jul 08 18:52:59.739183 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:59.739137 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c2e7ae4-1be9-450b-9d0b-52d16d2108b8-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bq554\" (UID: \"3c2e7ae4-1be9-450b-9d0b-52d16d2108b8\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bq554" Jul 08 18:52:59.739183 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:59.739155 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzstq\" (UniqueName: \"kubernetes.io/projected/3c2e7ae4-1be9-450b-9d0b-52d16d2108b8-kube-api-access-wzstq\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bq554\" (UID: \"3c2e7ae4-1be9-450b-9d0b-52d16d2108b8\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bq554" Jul 08 18:52:59.739533 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:59.739512 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c2e7ae4-1be9-450b-9d0b-52d16d2108b8-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bq554\" (UID: \"3c2e7ae4-1be9-450b-9d0b-52d16d2108b8\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bq554" Jul 08 18:52:59.739600 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:59.739584 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c2e7ae4-1be9-450b-9d0b-52d16d2108b8-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bq554\" (UID: \"3c2e7ae4-1be9-450b-9d0b-52d16d2108b8\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bq554" Jul 08 18:52:59.766905 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:59.766872 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzstq\" (UniqueName: \"kubernetes.io/projected/3c2e7ae4-1be9-450b-9d0b-52d16d2108b8-kube-api-access-wzstq\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bq554\" (UID: \"3c2e7ae4-1be9-450b-9d0b-52d16d2108b8\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bq554" Jul 08 18:52:59.786595 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:59.786564 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vbp48" Jul 08 18:52:59.907586 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:59.907553 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bq554" Jul 08 18:52:59.926610 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:59.926574 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfzkd"] Jul 08 18:52:59.931319 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:59.931295 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfzkd" Jul 08 18:52:59.941804 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:59.941751 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/666ebeb8-9b4e-4ec6-8f30-f27e760bf720-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfzkd\" (UID: \"666ebeb8-9b4e-4ec6-8f30-f27e760bf720\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfzkd" Jul 08 18:52:59.941952 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:59.941845 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g22m\" (UniqueName: \"kubernetes.io/projected/666ebeb8-9b4e-4ec6-8f30-f27e760bf720-kube-api-access-8g22m\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfzkd\" (UID: \"666ebeb8-9b4e-4ec6-8f30-f27e760bf720\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfzkd" Jul 08 18:52:59.941952 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:59.941887 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/666ebeb8-9b4e-4ec6-8f30-f27e760bf720-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfzkd\" (UID: \"666ebeb8-9b4e-4ec6-8f30-f27e760bf720\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfzkd" Jul 08 18:52:59.943380 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:59.943143 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfzkd"] Jul 08 18:52:59.943577 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:52:59.943480 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c0a620_41c0_4f36_ac2d_8a7f649b0216.slice/crio-a263f8e075d6937b23629ea00bfd62dfe4d6e92130f533fb09cec4016907fcb7 WatchSource:0}: Error finding container a263f8e075d6937b23629ea00bfd62dfe4d6e92130f533fb09cec4016907fcb7: Status 404 returned error can't find the container with id a263f8e075d6937b23629ea00bfd62dfe4d6e92130f533fb09cec4016907fcb7 Jul 08 18:52:59.945902 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:52:59.945836 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vbp48"] Jul 08 18:53:00.043372 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:00.043337 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g22m\" (UniqueName: \"kubernetes.io/projected/666ebeb8-9b4e-4ec6-8f30-f27e760bf720-kube-api-access-8g22m\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfzkd\" (UID: \"666ebeb8-9b4e-4ec6-8f30-f27e760bf720\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfzkd" Jul 08 18:53:00.043557 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:00.043384 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/666ebeb8-9b4e-4ec6-8f30-f27e760bf720-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfzkd\" (UID: \"666ebeb8-9b4e-4ec6-8f30-f27e760bf720\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfzkd" Jul 08 18:53:00.043557 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:00.043443 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/666ebeb8-9b4e-4ec6-8f30-f27e760bf720-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfzkd\" (UID: \"666ebeb8-9b4e-4ec6-8f30-f27e760bf720\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfzkd" Jul 08 18:53:00.043780 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:00.043759 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/666ebeb8-9b4e-4ec6-8f30-f27e760bf720-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfzkd\" (UID: \"666ebeb8-9b4e-4ec6-8f30-f27e760bf720\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfzkd" Jul 08 18:53:00.043856 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:00.043767 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/666ebeb8-9b4e-4ec6-8f30-f27e760bf720-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfzkd\" (UID: \"666ebeb8-9b4e-4ec6-8f30-f27e760bf720\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfzkd" Jul 08 18:53:00.052087 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:00.052063 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g22m\" (UniqueName: \"kubernetes.io/projected/666ebeb8-9b4e-4ec6-8f30-f27e760bf720-kube-api-access-8g22m\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfzkd\" (UID: \"666ebeb8-9b4e-4ec6-8f30-f27e760bf720\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfzkd" Jul 08 18:53:00.066199 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:00.066178 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bq554"] Jul 08 18:53:00.115012 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:53:00.114986 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c2e7ae4_1be9_450b_9d0b_52d16d2108b8.slice/crio-4f87b4a8b2f97e528c08ccc574ce5077da58893ff97a53d48bb11c4149b9f607 WatchSource:0}: Error finding container 4f87b4a8b2f97e528c08ccc574ce5077da58893ff97a53d48bb11c4149b9f607: Status 404 returned error can't find the container with id 4f87b4a8b2f97e528c08ccc574ce5077da58893ff97a53d48bb11c4149b9f607 Jul 08 18:53:00.232088 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:00.232054 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h27wj"] Jul 08 18:53:00.235558 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:00.235538 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h27wj" Jul 08 18:53:00.243291 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:00.243227 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfzkd" Jul 08 18:53:00.245563 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:00.245535 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/acf7738b-2bc0-41f9-862a-30e9b72ae410-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h27wj\" (UID: \"acf7738b-2bc0-41f9-862a-30e9b72ae410\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h27wj" Jul 08 18:53:00.245660 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:00.245623 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/acf7738b-2bc0-41f9-862a-30e9b72ae410-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h27wj\" (UID: \"acf7738b-2bc0-41f9-862a-30e9b72ae410\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h27wj" Jul 08 18:53:00.245709 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:00.245666 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvjkk\" (UniqueName: \"kubernetes.io/projected/acf7738b-2bc0-41f9-862a-30e9b72ae410-kube-api-access-qvjkk\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h27wj\" (UID: \"acf7738b-2bc0-41f9-862a-30e9b72ae410\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h27wj" Jul 08 18:53:00.254173 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:00.254153 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h27wj"] Jul 08 18:53:00.347078 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:00.347047 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/acf7738b-2bc0-41f9-862a-30e9b72ae410-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h27wj\" (UID: \"acf7738b-2bc0-41f9-862a-30e9b72ae410\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h27wj" Jul 08 18:53:00.347272 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:00.347136 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/acf7738b-2bc0-41f9-862a-30e9b72ae410-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h27wj\" (UID: \"acf7738b-2bc0-41f9-862a-30e9b72ae410\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h27wj" Jul 08 18:53:00.347272 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:00.347180 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvjkk\" (UniqueName: \"kubernetes.io/projected/acf7738b-2bc0-41f9-862a-30e9b72ae410-kube-api-access-qvjkk\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h27wj\" (UID: \"acf7738b-2bc0-41f9-862a-30e9b72ae410\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h27wj" Jul 08 18:53:00.347524 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:00.347500 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/acf7738b-2bc0-41f9-862a-30e9b72ae410-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h27wj\" (UID: \"acf7738b-2bc0-41f9-862a-30e9b72ae410\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h27wj" Jul 08 18:53:00.347595 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:00.347527 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/acf7738b-2bc0-41f9-862a-30e9b72ae410-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h27wj\" (UID: \"acf7738b-2bc0-41f9-862a-30e9b72ae410\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h27wj" Jul 08 18:53:00.359300 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:00.359274 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvjkk\" (UniqueName: \"kubernetes.io/projected/acf7738b-2bc0-41f9-862a-30e9b72ae410-kube-api-access-qvjkk\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h27wj\" (UID: \"acf7738b-2bc0-41f9-862a-30e9b72ae410\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h27wj" Jul 08 18:53:00.381004 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:00.380983 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfzkd"] Jul 08 18:53:00.383093 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:53:00.383069 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod666ebeb8_9b4e_4ec6_8f30_f27e760bf720.slice/crio-37029865133b658e4438bcefb83bafae34aed3fc5a9936300497c672ab8fff00 WatchSource:0}: Error finding container 37029865133b658e4438bcefb83bafae34aed3fc5a9936300497c672ab8fff00: Status 404 returned error can't find the container with id 37029865133b658e4438bcefb83bafae34aed3fc5a9936300497c672ab8fff00 Jul 08 18:53:00.579050 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:00.578974 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h27wj" Jul 08 18:53:00.680710 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:00.680677 2570 generic.go:356] "Generic (PLEG): container finished" podID="666ebeb8-9b4e-4ec6-8f30-f27e760bf720" containerID="59951e53390039c7f947171d7ad64e5a6d92fd160c81c13b165fac99aa47c24c" exitCode=0 Jul 08 18:53:00.680964 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:00.680770 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfzkd" event={"ID":"666ebeb8-9b4e-4ec6-8f30-f27e760bf720","Type":"ContainerDied","Data":"59951e53390039c7f947171d7ad64e5a6d92fd160c81c13b165fac99aa47c24c"} Jul 08 18:53:00.680964 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:00.680827 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfzkd" event={"ID":"666ebeb8-9b4e-4ec6-8f30-f27e760bf720","Type":"ContainerStarted","Data":"37029865133b658e4438bcefb83bafae34aed3fc5a9936300497c672ab8fff00"} Jul 08 18:53:00.683009 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:00.682979 2570 generic.go:356] "Generic (PLEG): container finished" podID="3c2e7ae4-1be9-450b-9d0b-52d16d2108b8" containerID="2388e89108d9d969ad2f8970ca80cfd23c51da8a85f619e43f3d98d4bb65f7f2" exitCode=0 Jul 08 18:53:00.683120 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:00.683078 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bq554" event={"ID":"3c2e7ae4-1be9-450b-9d0b-52d16d2108b8","Type":"ContainerDied","Data":"2388e89108d9d969ad2f8970ca80cfd23c51da8a85f619e43f3d98d4bb65f7f2"} Jul 08 18:53:00.683188 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:00.683129 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bq554" event={"ID":"3c2e7ae4-1be9-450b-9d0b-52d16d2108b8","Type":"ContainerStarted","Data":"4f87b4a8b2f97e528c08ccc574ce5077da58893ff97a53d48bb11c4149b9f607"} Jul 08 18:53:00.690128 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:00.690091 2570 generic.go:356] "Generic (PLEG): container finished" podID="55c0a620-41c0-4f36-ac2d-8a7f649b0216" containerID="5904749cd8b553c9f29b1e7877ba99b4fc625393a2d3618ab4db43efb0bded73" exitCode=0 Jul 08 18:53:00.690295 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:00.690154 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vbp48" event={"ID":"55c0a620-41c0-4f36-ac2d-8a7f649b0216","Type":"ContainerDied","Data":"5904749cd8b553c9f29b1e7877ba99b4fc625393a2d3618ab4db43efb0bded73"} Jul 08 18:53:00.690295 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:00.690193 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vbp48" event={"ID":"55c0a620-41c0-4f36-ac2d-8a7f649b0216","Type":"ContainerStarted","Data":"a263f8e075d6937b23629ea00bfd62dfe4d6e92130f533fb09cec4016907fcb7"} Jul 08 18:53:00.740308 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:00.740277 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h27wj"] Jul 08 18:53:00.743307 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:53:00.743277 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacf7738b_2bc0_41f9_862a_30e9b72ae410.slice/crio-78d9a009868419935ff5828f2b61db982085714a588cf1fcf8b58a464a04a63d WatchSource:0}: Error finding container 78d9a009868419935ff5828f2b61db982085714a588cf1fcf8b58a464a04a63d: Status 404 returned error can't find the container with id 78d9a009868419935ff5828f2b61db982085714a588cf1fcf8b58a464a04a63d Jul 08 18:53:01.697116 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:01.697085 2570 generic.go:356] "Generic (PLEG): container finished" podID="55c0a620-41c0-4f36-ac2d-8a7f649b0216" containerID="a17fac2ee1e6ba0890585481bea71a6b8467f07331fbfe54709cb5baf54d8a2a" exitCode=0 Jul 08 18:53:01.697577 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:01.697114 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vbp48" event={"ID":"55c0a620-41c0-4f36-ac2d-8a7f649b0216","Type":"ContainerDied","Data":"a17fac2ee1e6ba0890585481bea71a6b8467f07331fbfe54709cb5baf54d8a2a"} Jul 08 18:53:01.698840 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:01.698821 2570 generic.go:356] "Generic (PLEG): container finished" podID="acf7738b-2bc0-41f9-862a-30e9b72ae410" containerID="1e8c64403739456394d9e6c0a3b9a2fd8e5bc883ffa02acea236880937d11611" exitCode=0 Jul 08 18:53:01.698957 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:01.698852 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h27wj" event={"ID":"acf7738b-2bc0-41f9-862a-30e9b72ae410","Type":"ContainerDied","Data":"1e8c64403739456394d9e6c0a3b9a2fd8e5bc883ffa02acea236880937d11611"} Jul 08 18:53:01.698957 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:01.698887 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h27wj" event={"ID":"acf7738b-2bc0-41f9-862a-30e9b72ae410","Type":"ContainerStarted","Data":"78d9a009868419935ff5828f2b61db982085714a588cf1fcf8b58a464a04a63d"} Jul 08 18:53:02.706563 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:02.706533 2570 generic.go:356] "Generic (PLEG): container finished" podID="666ebeb8-9b4e-4ec6-8f30-f27e760bf720" containerID="343f3a24807a19caa2f65a62b8720e8fd62283c4eaac9dc7f8a49ce61264cbc5" exitCode=0 Jul 08 18:53:02.706919 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:02.706618 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfzkd" event={"ID":"666ebeb8-9b4e-4ec6-8f30-f27e760bf720","Type":"ContainerDied","Data":"343f3a24807a19caa2f65a62b8720e8fd62283c4eaac9dc7f8a49ce61264cbc5"} Jul 08 18:53:02.708501 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:02.708482 2570 generic.go:356] "Generic (PLEG): container finished" podID="3c2e7ae4-1be9-450b-9d0b-52d16d2108b8" containerID="14b431326f2c6ba0a251cb72a545acb656a4c16d674807c8b020bffca301ffc9" exitCode=0 Jul 08 18:53:02.708608 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:02.708539 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bq554" event={"ID":"3c2e7ae4-1be9-450b-9d0b-52d16d2108b8","Type":"ContainerDied","Data":"14b431326f2c6ba0a251cb72a545acb656a4c16d674807c8b020bffca301ffc9"} Jul 08 18:53:02.710777 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:02.710756 2570 generic.go:356] "Generic (PLEG): container finished" podID="55c0a620-41c0-4f36-ac2d-8a7f649b0216" containerID="4ed5454203e6be788b29ea755bfb5278906c4f2675ea8661d4e999ba13b8f349" exitCode=0 Jul 08 18:53:02.710890 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:02.710833 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vbp48" event={"ID":"55c0a620-41c0-4f36-ac2d-8a7f649b0216","Type":"ContainerDied","Data":"4ed5454203e6be788b29ea755bfb5278906c4f2675ea8661d4e999ba13b8f349"} Jul 08 18:53:02.712763 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:02.712740 2570 generic.go:356] "Generic (PLEG): container finished" podID="acf7738b-2bc0-41f9-862a-30e9b72ae410" containerID="f80b26c78e6d688effe0bb5f37ef5afd6f88513b1f542fcc31c802e1f7f4efbe" exitCode=0 Jul 08 18:53:02.712866 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:02.712779 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h27wj" event={"ID":"acf7738b-2bc0-41f9-862a-30e9b72ae410","Type":"ContainerDied","Data":"f80b26c78e6d688effe0bb5f37ef5afd6f88513b1f542fcc31c802e1f7f4efbe"} Jul 08 18:53:03.721248 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:03.721212 2570 generic.go:356] "Generic (PLEG): container finished" podID="acf7738b-2bc0-41f9-862a-30e9b72ae410" containerID="51f2e08fbd7a7563bf98008e40001fa6f663d0b0adaaa4774f732c4c58c2a221" exitCode=0 Jul 08 18:53:03.721846 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:03.721287 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h27wj" event={"ID":"acf7738b-2bc0-41f9-862a-30e9b72ae410","Type":"ContainerDied","Data":"51f2e08fbd7a7563bf98008e40001fa6f663d0b0adaaa4774f732c4c58c2a221"} Jul 08 18:53:03.723783 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:03.723760 2570 generic.go:356] "Generic (PLEG): container finished" podID="666ebeb8-9b4e-4ec6-8f30-f27e760bf720" containerID="4a35846f31d8a5e63029834a21b147ab2585d1772f9c40fb1d4d8cd1a040fc1b" exitCode=0 Jul 08 18:53:03.723905 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:03.723823 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfzkd" event={"ID":"666ebeb8-9b4e-4ec6-8f30-f27e760bf720","Type":"ContainerDied","Data":"4a35846f31d8a5e63029834a21b147ab2585d1772f9c40fb1d4d8cd1a040fc1b"} Jul 08 18:53:03.726163 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:03.726144 2570 generic.go:356] "Generic (PLEG): container finished" podID="3c2e7ae4-1be9-450b-9d0b-52d16d2108b8" containerID="db0dc47c95a6fbc25e8e75185d68becc5b5d06d2eb7b35c6c4f4ee7a56b0c758" exitCode=0 Jul 08 18:53:03.726297 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:03.726228 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bq554" event={"ID":"3c2e7ae4-1be9-450b-9d0b-52d16d2108b8","Type":"ContainerDied","Data":"db0dc47c95a6fbc25e8e75185d68becc5b5d06d2eb7b35c6c4f4ee7a56b0c758"} Jul 08 18:53:03.860315 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:03.860291 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vbp48" Jul 08 18:53:03.880262 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:03.880225 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/55c0a620-41c0-4f36-ac2d-8a7f649b0216-util\") pod \"55c0a620-41c0-4f36-ac2d-8a7f649b0216\" (UID: \"55c0a620-41c0-4f36-ac2d-8a7f649b0216\") " Jul 08 18:53:03.880370 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:03.880354 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwq5r\" (UniqueName: \"kubernetes.io/projected/55c0a620-41c0-4f36-ac2d-8a7f649b0216-kube-api-access-zwq5r\") pod \"55c0a620-41c0-4f36-ac2d-8a7f649b0216\" (UID: \"55c0a620-41c0-4f36-ac2d-8a7f649b0216\") " Jul 08 18:53:03.880410 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:03.880389 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/55c0a620-41c0-4f36-ac2d-8a7f649b0216-bundle\") pod \"55c0a620-41c0-4f36-ac2d-8a7f649b0216\" (UID: \"55c0a620-41c0-4f36-ac2d-8a7f649b0216\") " Jul 08 18:53:03.881151 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:03.881113 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c0a620-41c0-4f36-ac2d-8a7f649b0216-bundle" (OuterVolumeSpecName: "bundle") pod "55c0a620-41c0-4f36-ac2d-8a7f649b0216" (UID: "55c0a620-41c0-4f36-ac2d-8a7f649b0216"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:53:03.882669 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:03.882640 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c0a620-41c0-4f36-ac2d-8a7f649b0216-kube-api-access-zwq5r" (OuterVolumeSpecName: "kube-api-access-zwq5r") pod "55c0a620-41c0-4f36-ac2d-8a7f649b0216" (UID: "55c0a620-41c0-4f36-ac2d-8a7f649b0216"). InnerVolumeSpecName "kube-api-access-zwq5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:53:03.887219 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:03.887193 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c0a620-41c0-4f36-ac2d-8a7f649b0216-util" (OuterVolumeSpecName: "util") pod "55c0a620-41c0-4f36-ac2d-8a7f649b0216" (UID: "55c0a620-41c0-4f36-ac2d-8a7f649b0216"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:53:03.981823 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:03.981758 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwq5r\" (UniqueName: \"kubernetes.io/projected/55c0a620-41c0-4f36-ac2d-8a7f649b0216-kube-api-access-zwq5r\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:53:03.981823 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:03.981781 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/55c0a620-41c0-4f36-ac2d-8a7f649b0216-bundle\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:53:03.981823 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:03.981790 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/55c0a620-41c0-4f36-ac2d-8a7f649b0216-util\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:53:04.733764 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:04.733740 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vbp48" Jul 08 18:53:04.734136 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:04.733787 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vbp48" event={"ID":"55c0a620-41c0-4f36-ac2d-8a7f649b0216","Type":"ContainerDied","Data":"a263f8e075d6937b23629ea00bfd62dfe4d6e92130f533fb09cec4016907fcb7"} Jul 08 18:53:04.734136 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:04.733830 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a263f8e075d6937b23629ea00bfd62dfe4d6e92130f533fb09cec4016907fcb7" Jul 08 18:53:04.893195 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:04.893171 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h27wj" Jul 08 18:53:04.896785 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:04.896764 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bq554" Jul 08 18:53:04.924807 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:04.924784 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfzkd" Jul 08 18:53:04.990654 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:04.990578 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/acf7738b-2bc0-41f9-862a-30e9b72ae410-bundle\") pod \"acf7738b-2bc0-41f9-862a-30e9b72ae410\" (UID: \"acf7738b-2bc0-41f9-862a-30e9b72ae410\") " Jul 08 18:53:04.990654 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:04.990636 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/666ebeb8-9b4e-4ec6-8f30-f27e760bf720-bundle\") pod \"666ebeb8-9b4e-4ec6-8f30-f27e760bf720\" (UID: \"666ebeb8-9b4e-4ec6-8f30-f27e760bf720\") " Jul 08 18:53:04.990870 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:04.990678 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c2e7ae4-1be9-450b-9d0b-52d16d2108b8-util\") pod \"3c2e7ae4-1be9-450b-9d0b-52d16d2108b8\" (UID: \"3c2e7ae4-1be9-450b-9d0b-52d16d2108b8\") " Jul 08 18:53:04.990870 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:04.990717 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzstq\" (UniqueName: \"kubernetes.io/projected/3c2e7ae4-1be9-450b-9d0b-52d16d2108b8-kube-api-access-wzstq\") pod \"3c2e7ae4-1be9-450b-9d0b-52d16d2108b8\" (UID: \"3c2e7ae4-1be9-450b-9d0b-52d16d2108b8\") " Jul 08 18:53:04.990870 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:04.990773 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8g22m\" (UniqueName: \"kubernetes.io/projected/666ebeb8-9b4e-4ec6-8f30-f27e760bf720-kube-api-access-8g22m\") pod \"666ebeb8-9b4e-4ec6-8f30-f27e760bf720\" (UID: \"666ebeb8-9b4e-4ec6-8f30-f27e760bf720\") " Jul 08 18:53:04.990870 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:04.990854 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c2e7ae4-1be9-450b-9d0b-52d16d2108b8-bundle\") pod \"3c2e7ae4-1be9-450b-9d0b-52d16d2108b8\" (UID: \"3c2e7ae4-1be9-450b-9d0b-52d16d2108b8\") " Jul 08 18:53:04.991073 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:04.990878 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/666ebeb8-9b4e-4ec6-8f30-f27e760bf720-util\") pod \"666ebeb8-9b4e-4ec6-8f30-f27e760bf720\" (UID: \"666ebeb8-9b4e-4ec6-8f30-f27e760bf720\") " Jul 08 18:53:04.991073 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:04.990919 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/acf7738b-2bc0-41f9-862a-30e9b72ae410-util\") pod \"acf7738b-2bc0-41f9-862a-30e9b72ae410\" (UID: \"acf7738b-2bc0-41f9-862a-30e9b72ae410\") " Jul 08 18:53:04.991073 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:04.990948 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvjkk\" (UniqueName: \"kubernetes.io/projected/acf7738b-2bc0-41f9-862a-30e9b72ae410-kube-api-access-qvjkk\") pod \"acf7738b-2bc0-41f9-862a-30e9b72ae410\" (UID: \"acf7738b-2bc0-41f9-862a-30e9b72ae410\") " Jul 08 18:53:04.991201 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:04.991112 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acf7738b-2bc0-41f9-862a-30e9b72ae410-bundle" (OuterVolumeSpecName: "bundle") pod "acf7738b-2bc0-41f9-862a-30e9b72ae410" (UID: "acf7738b-2bc0-41f9-862a-30e9b72ae410"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:53:04.991250 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:04.991220 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/acf7738b-2bc0-41f9-862a-30e9b72ae410-bundle\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:53:04.991447 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:04.991372 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666ebeb8-9b4e-4ec6-8f30-f27e760bf720-bundle" (OuterVolumeSpecName: "bundle") pod "666ebeb8-9b4e-4ec6-8f30-f27e760bf720" (UID: "666ebeb8-9b4e-4ec6-8f30-f27e760bf720"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:53:04.991963 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:04.991931 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2e7ae4-1be9-450b-9d0b-52d16d2108b8-bundle" (OuterVolumeSpecName: "bundle") pod "3c2e7ae4-1be9-450b-9d0b-52d16d2108b8" (UID: "3c2e7ae4-1be9-450b-9d0b-52d16d2108b8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:53:04.994117 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:04.994089 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/666ebeb8-9b4e-4ec6-8f30-f27e760bf720-kube-api-access-8g22m" (OuterVolumeSpecName: "kube-api-access-8g22m") pod "666ebeb8-9b4e-4ec6-8f30-f27e760bf720" (UID: "666ebeb8-9b4e-4ec6-8f30-f27e760bf720"). InnerVolumeSpecName "kube-api-access-8g22m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:53:04.994210 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:04.994198 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acf7738b-2bc0-41f9-862a-30e9b72ae410-kube-api-access-qvjkk" (OuterVolumeSpecName: "kube-api-access-qvjkk") pod "acf7738b-2bc0-41f9-862a-30e9b72ae410" (UID: "acf7738b-2bc0-41f9-862a-30e9b72ae410"). InnerVolumeSpecName "kube-api-access-qvjkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:53:04.994299 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:04.994274 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c2e7ae4-1be9-450b-9d0b-52d16d2108b8-kube-api-access-wzstq" (OuterVolumeSpecName: "kube-api-access-wzstq") pod "3c2e7ae4-1be9-450b-9d0b-52d16d2108b8" (UID: "3c2e7ae4-1be9-450b-9d0b-52d16d2108b8"). InnerVolumeSpecName "kube-api-access-wzstq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:53:04.998390 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:04.998363 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666ebeb8-9b4e-4ec6-8f30-f27e760bf720-util" (OuterVolumeSpecName: "util") pod "666ebeb8-9b4e-4ec6-8f30-f27e760bf720" (UID: "666ebeb8-9b4e-4ec6-8f30-f27e760bf720"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:53:04.998667 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:04.998646 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acf7738b-2bc0-41f9-862a-30e9b72ae410-util" (OuterVolumeSpecName: "util") pod "acf7738b-2bc0-41f9-862a-30e9b72ae410" (UID: "acf7738b-2bc0-41f9-862a-30e9b72ae410"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:53:04.998724 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:04.998661 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2e7ae4-1be9-450b-9d0b-52d16d2108b8-util" (OuterVolumeSpecName: "util") pod "3c2e7ae4-1be9-450b-9d0b-52d16d2108b8" (UID: "3c2e7ae4-1be9-450b-9d0b-52d16d2108b8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:53:05.091734 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:05.091703 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g22m\" (UniqueName: \"kubernetes.io/projected/666ebeb8-9b4e-4ec6-8f30-f27e760bf720-kube-api-access-8g22m\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:53:05.091734 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:05.091726 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c2e7ae4-1be9-450b-9d0b-52d16d2108b8-bundle\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:53:05.091734 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:05.091737 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/666ebeb8-9b4e-4ec6-8f30-f27e760bf720-util\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:53:05.091943 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:05.091745 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/acf7738b-2bc0-41f9-862a-30e9b72ae410-util\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:53:05.091943 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:05.091753 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvjkk\" (UniqueName: \"kubernetes.io/projected/acf7738b-2bc0-41f9-862a-30e9b72ae410-kube-api-access-qvjkk\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:53:05.091943 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:05.091763 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/666ebeb8-9b4e-4ec6-8f30-f27e760bf720-bundle\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:53:05.091943 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:05.091773 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c2e7ae4-1be9-450b-9d0b-52d16d2108b8-util\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:53:05.091943 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:05.091781 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzstq\" (UniqueName: \"kubernetes.io/projected/3c2e7ae4-1be9-450b-9d0b-52d16d2108b8-kube-api-access-wzstq\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:53:05.740711 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:05.740680 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bq554" Jul 08 18:53:05.741095 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:05.740705 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bq554" event={"ID":"3c2e7ae4-1be9-450b-9d0b-52d16d2108b8","Type":"ContainerDied","Data":"4f87b4a8b2f97e528c08ccc574ce5077da58893ff97a53d48bb11c4149b9f607"} Jul 08 18:53:05.741095 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:05.740765 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f87b4a8b2f97e528c08ccc574ce5077da58893ff97a53d48bb11c4149b9f607" Jul 08 18:53:05.742872 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:05.742848 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h27wj" Jul 08 18:53:05.742998 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:05.742876 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h27wj" event={"ID":"acf7738b-2bc0-41f9-862a-30e9b72ae410","Type":"ContainerDied","Data":"78d9a009868419935ff5828f2b61db982085714a588cf1fcf8b58a464a04a63d"} Jul 08 18:53:05.742998 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:05.742918 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78d9a009868419935ff5828f2b61db982085714a588cf1fcf8b58a464a04a63d" Jul 08 18:53:05.745046 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:05.745024 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfzkd" event={"ID":"666ebeb8-9b4e-4ec6-8f30-f27e760bf720","Type":"ContainerDied","Data":"37029865133b658e4438bcefb83bafae34aed3fc5a9936300497c672ab8fff00"} Jul 08 18:53:05.745164 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:05.745063 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37029865133b658e4438bcefb83bafae34aed3fc5a9936300497c672ab8fff00" Jul 08 18:53:05.745164 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:05.745035 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfzkd" Jul 08 18:53:10.629367 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.629334 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-br2mx"] Jul 08 18:53:10.629728 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.629662 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="666ebeb8-9b4e-4ec6-8f30-f27e760bf720" containerName="util" Jul 08 18:53:10.629728 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.629675 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="666ebeb8-9b4e-4ec6-8f30-f27e760bf720" containerName="util" Jul 08 18:53:10.629728 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.629684 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55c0a620-41c0-4f36-ac2d-8a7f649b0216" containerName="util" Jul 08 18:53:10.629728 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.629690 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="55c0a620-41c0-4f36-ac2d-8a7f649b0216" containerName="util" Jul 08 18:53:10.629728 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.629695 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c2e7ae4-1be9-450b-9d0b-52d16d2108b8" containerName="pull" Jul 08 18:53:10.629728 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.629700 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c2e7ae4-1be9-450b-9d0b-52d16d2108b8" containerName="pull" Jul 08 18:53:10.629728 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.629709 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acf7738b-2bc0-41f9-862a-30e9b72ae410" containerName="extract" Jul 08 18:53:10.629728 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.629714 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="acf7738b-2bc0-41f9-862a-30e9b72ae410" containerName="extract" Jul 08 18:53:10.629728 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.629725 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c2e7ae4-1be9-450b-9d0b-52d16d2108b8" containerName="extract" Jul 08 18:53:10.629728 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.629730 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c2e7ae4-1be9-450b-9d0b-52d16d2108b8" containerName="extract" Jul 08 18:53:10.630061 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.629737 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acf7738b-2bc0-41f9-862a-30e9b72ae410" containerName="util" Jul 08 18:53:10.630061 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.629742 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="acf7738b-2bc0-41f9-862a-30e9b72ae410" containerName="util" Jul 08 18:53:10.630061 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.629751 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55c0a620-41c0-4f36-ac2d-8a7f649b0216" containerName="extract" Jul 08 18:53:10.630061 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.629756 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="55c0a620-41c0-4f36-ac2d-8a7f649b0216" containerName="extract" Jul 08 18:53:10.630061 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.629762 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="666ebeb8-9b4e-4ec6-8f30-f27e760bf720" containerName="pull" Jul 08 18:53:10.630061 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.629767 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="666ebeb8-9b4e-4ec6-8f30-f27e760bf720" containerName="pull" Jul 08 18:53:10.630061 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.629775 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55c0a620-41c0-4f36-ac2d-8a7f649b0216" containerName="pull" Jul 08 18:53:10.630061 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.629779 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="55c0a620-41c0-4f36-ac2d-8a7f649b0216" containerName="pull" Jul 08 18:53:10.630061 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.629784 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="666ebeb8-9b4e-4ec6-8f30-f27e760bf720" containerName="extract" Jul 08 18:53:10.630061 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.629789 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="666ebeb8-9b4e-4ec6-8f30-f27e760bf720" containerName="extract" Jul 08 18:53:10.630061 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.629794 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c2e7ae4-1be9-450b-9d0b-52d16d2108b8" containerName="util" Jul 08 18:53:10.630061 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.629799 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c2e7ae4-1be9-450b-9d0b-52d16d2108b8" containerName="util" Jul 08 18:53:10.630061 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.629806 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acf7738b-2bc0-41f9-862a-30e9b72ae410" containerName="pull" Jul 08 18:53:10.630061 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.629811 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="acf7738b-2bc0-41f9-862a-30e9b72ae410" containerName="pull" Jul 08 18:53:10.630061 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.629859 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="55c0a620-41c0-4f36-ac2d-8a7f649b0216" containerName="extract" Jul 08 18:53:10.630061 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.629867 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="666ebeb8-9b4e-4ec6-8f30-f27e760bf720" containerName="extract" Jul 08 18:53:10.630061 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.629872 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c2e7ae4-1be9-450b-9d0b-52d16d2108b8" containerName="extract" Jul 08 18:53:10.630061 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.629881 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="acf7738b-2bc0-41f9-862a-30e9b72ae410" containerName="extract" Jul 08 18:53:10.633193 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.633175 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-br2mx" Jul 08 18:53:10.640957 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.640936 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 08 18:53:10.641069 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.640982 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-vs4bf\"" Jul 08 18:53:10.690826 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.690801 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-br2mx"] Jul 08 18:53:10.738331 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.738302 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlxqb\" (UniqueName: \"kubernetes.io/projected/1ea34d8d-e04b-4fad-b8f7-f48449f8e1d3-kube-api-access-vlxqb\") pod \"dns-operator-controller-manager-864dd78c4c-br2mx\" (UID: \"1ea34d8d-e04b-4fad-b8f7-f48449f8e1d3\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-br2mx" Jul 08 18:53:10.839408 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.839384 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlxqb\" (UniqueName: \"kubernetes.io/projected/1ea34d8d-e04b-4fad-b8f7-f48449f8e1d3-kube-api-access-vlxqb\") pod \"dns-operator-controller-manager-864dd78c4c-br2mx\" (UID: \"1ea34d8d-e04b-4fad-b8f7-f48449f8e1d3\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-br2mx" Jul 08 18:53:10.861477 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.861454 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlxqb\" (UniqueName: \"kubernetes.io/projected/1ea34d8d-e04b-4fad-b8f7-f48449f8e1d3-kube-api-access-vlxqb\") pod \"dns-operator-controller-manager-864dd78c4c-br2mx\" (UID: \"1ea34d8d-e04b-4fad-b8f7-f48449f8e1d3\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-br2mx" Jul 08 18:53:10.942415 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:10.942351 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-br2mx" Jul 08 18:53:11.123516 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:11.123491 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-br2mx"] Jul 08 18:53:11.124963 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:53:11.124937 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ea34d8d_e04b_4fad_b8f7_f48449f8e1d3.slice/crio-aede52561ca8b251a5e05ed587e8a04f15a327661969f80ab050554102a61b89 WatchSource:0}: Error finding container aede52561ca8b251a5e05ed587e8a04f15a327661969f80ab050554102a61b89: Status 404 returned error can't find the container with id aede52561ca8b251a5e05ed587e8a04f15a327661969f80ab050554102a61b89 Jul 08 18:53:11.777165 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:11.777132 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-br2mx" event={"ID":"1ea34d8d-e04b-4fad-b8f7-f48449f8e1d3","Type":"ContainerStarted","Data":"aede52561ca8b251a5e05ed587e8a04f15a327661969f80ab050554102a61b89"} Jul 08 18:53:14.798227 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:14.798192 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-br2mx" event={"ID":"1ea34d8d-e04b-4fad-b8f7-f48449f8e1d3","Type":"ContainerStarted","Data":"2d30950c59659eaebb7080eb25158ebd7c761ab50cd2e38671448b93bd395647"} Jul 08 18:53:14.798626 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:14.798312 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-br2mx" Jul 08 18:53:14.824026 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:14.823975 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-br2mx" podStartSLOduration=2.180081508 podStartE2EDuration="4.823963s" totalImagesPullingTime="2.643881492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:53:14.822007618 +0000 UTC m=+455.669282283" watchObservedRunningTime="2026-07-08 18:53:14.823963 +0000 UTC m=+455.671237667" Jul 08 18:53:14.907583 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:14.907556 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-qsplg"] Jul 08 18:53:14.910934 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:14.910921 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qsplg" Jul 08 18:53:14.914007 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:14.913989 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-gfprp\"" Jul 08 18:53:14.921582 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:14.921558 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-qsplg"] Jul 08 18:53:14.974474 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:14.974448 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fqbv\" (UniqueName: \"kubernetes.io/projected/4f2ed85e-3619-441c-ac5d-f1608dcec2c3-kube-api-access-7fqbv\") pod \"authorino-operator-75cc8d5cb7-qsplg\" (UID: \"4f2ed85e-3619-441c-ac5d-f1608dcec2c3\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-qsplg" Jul 08 18:53:15.075206 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:15.075112 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fqbv\" (UniqueName: \"kubernetes.io/projected/4f2ed85e-3619-441c-ac5d-f1608dcec2c3-kube-api-access-7fqbv\") pod \"authorino-operator-75cc8d5cb7-qsplg\" (UID: \"4f2ed85e-3619-441c-ac5d-f1608dcec2c3\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-qsplg" Jul 08 18:53:15.090428 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:15.090399 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fqbv\" (UniqueName: \"kubernetes.io/projected/4f2ed85e-3619-441c-ac5d-f1608dcec2c3-kube-api-access-7fqbv\") pod \"authorino-operator-75cc8d5cb7-qsplg\" (UID: \"4f2ed85e-3619-441c-ac5d-f1608dcec2c3\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-qsplg" Jul 08 18:53:15.221478 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:15.221442 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qsplg" Jul 08 18:53:15.358512 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:15.358490 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-qsplg"] Jul 08 18:53:15.360115 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:53:15.360091 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f2ed85e_3619_441c_ac5d_f1608dcec2c3.slice/crio-fbc37e3473770ceaf6d5b828ebf07be7f20e48cfd265ade2332e0ad1e4eb7aac WatchSource:0}: Error finding container fbc37e3473770ceaf6d5b828ebf07be7f20e48cfd265ade2332e0ad1e4eb7aac: Status 404 returned error can't find the container with id fbc37e3473770ceaf6d5b828ebf07be7f20e48cfd265ade2332e0ad1e4eb7aac Jul 08 18:53:15.806395 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:15.806343 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qsplg" event={"ID":"4f2ed85e-3619-441c-ac5d-f1608dcec2c3","Type":"ContainerStarted","Data":"fbc37e3473770ceaf6d5b828ebf07be7f20e48cfd265ade2332e0ad1e4eb7aac"} Jul 08 18:53:17.821189 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:17.821162 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qsplg" event={"ID":"4f2ed85e-3619-441c-ac5d-f1608dcec2c3","Type":"ContainerStarted","Data":"38c70f751b7091e6e0e82f50c346e9857977b272a4be62d3ba6f43164cf9d1de"} Jul 08 18:53:17.821496 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:17.821241 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qsplg" Jul 08 18:53:17.842908 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:17.842858 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qsplg" podStartSLOduration=1.46967365 podStartE2EDuration="3.842841258s" totalImagesPullingTime="2.373167608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:53:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:53:17.841895051 +0000 UTC m=+458.689169721" watchObservedRunningTime="2026-07-08 18:53:17.842841258 +0000 UTC m=+458.690115924" Jul 08 18:53:18.090530 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:18.090456 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rslwk"] Jul 08 18:53:18.093899 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:18.093866 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rslwk" Jul 08 18:53:18.096616 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:18.096593 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-tsnfp\"" Jul 08 18:53:18.110541 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:18.110519 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rslwk"] Jul 08 18:53:18.202547 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:18.202513 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8pr9\" (UniqueName: \"kubernetes.io/projected/860f6fef-88d8-4642-bbab-d889138c344e-kube-api-access-t8pr9\") pod \"limitador-operator-controller-manager-859d7b78bd-rslwk\" (UID: \"860f6fef-88d8-4642-bbab-d889138c344e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rslwk" Jul 08 18:53:18.303858 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:18.303828 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8pr9\" (UniqueName: \"kubernetes.io/projected/860f6fef-88d8-4642-bbab-d889138c344e-kube-api-access-t8pr9\") pod \"limitador-operator-controller-manager-859d7b78bd-rslwk\" (UID: \"860f6fef-88d8-4642-bbab-d889138c344e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rslwk" Jul 08 18:53:18.312181 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:18.312154 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8pr9\" (UniqueName: \"kubernetes.io/projected/860f6fef-88d8-4642-bbab-d889138c344e-kube-api-access-t8pr9\") pod \"limitador-operator-controller-manager-859d7b78bd-rslwk\" (UID: \"860f6fef-88d8-4642-bbab-d889138c344e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rslwk" Jul 08 18:53:18.404801 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:18.404773 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rslwk" Jul 08 18:53:18.553481 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:18.553452 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rslwk"] Jul 08 18:53:18.553604 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:53:18.553571 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod860f6fef_88d8_4642_bbab_d889138c344e.slice/crio-f66cecc5b215fd0199c97b13530dfb136dedceba98ae20684d1625ed7189cda0 WatchSource:0}: Error finding container f66cecc5b215fd0199c97b13530dfb136dedceba98ae20684d1625ed7189cda0: Status 404 returned error can't find the container with id f66cecc5b215fd0199c97b13530dfb136dedceba98ae20684d1625ed7189cda0 Jul 08 18:53:18.827855 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:18.827765 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rslwk" event={"ID":"860f6fef-88d8-4642-bbab-d889138c344e","Type":"ContainerStarted","Data":"f66cecc5b215fd0199c97b13530dfb136dedceba98ae20684d1625ed7189cda0"} Jul 08 18:53:20.840697 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:20.840649 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rslwk" event={"ID":"860f6fef-88d8-4642-bbab-d889138c344e","Type":"ContainerStarted","Data":"324cd9d8730afd6f34922dfa02ff93675ff1d8fdc196c6b04ab40ef160dabd0d"} Jul 08 18:53:20.841128 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:20.840827 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rslwk" Jul 08 18:53:20.862458 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:20.862409 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rslwk" podStartSLOduration=0.776582399 podStartE2EDuration="2.862395102s" totalImagesPullingTime="2.085812703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:53:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:53:20.859986068 +0000 UTC m=+461.707260734" watchObservedRunningTime="2026-07-08 18:53:20.862395102 +0000 UTC m=+461.709669768" Jul 08 18:53:22.832761 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:22.832724 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pq6d6"] Jul 08 18:53:22.836390 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:22.836371 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pq6d6" Jul 08 18:53:22.839328 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:22.839307 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rxz88\"" Jul 08 18:53:22.853367 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:22.853344 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pq6d6"] Jul 08 18:53:22.946150 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:22.946115 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m7wg\" (UniqueName: \"kubernetes.io/projected/d8807004-fc1f-460f-abd0-5af28c9948d5-kube-api-access-8m7wg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-pq6d6\" (UID: \"d8807004-fc1f-460f-abd0-5af28c9948d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pq6d6" Jul 08 18:53:22.946333 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:22.946173 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8807004-fc1f-460f-abd0-5af28c9948d5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-pq6d6\" (UID: \"d8807004-fc1f-460f-abd0-5af28c9948d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pq6d6" Jul 08 18:53:23.047180 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:23.047143 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m7wg\" (UniqueName: \"kubernetes.io/projected/d8807004-fc1f-460f-abd0-5af28c9948d5-kube-api-access-8m7wg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-pq6d6\" (UID: \"d8807004-fc1f-460f-abd0-5af28c9948d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pq6d6" Jul 08 18:53:23.047404 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:23.047205 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8807004-fc1f-460f-abd0-5af28c9948d5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-pq6d6\" (UID: \"d8807004-fc1f-460f-abd0-5af28c9948d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pq6d6" Jul 08 18:53:23.047662 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:23.047643 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8807004-fc1f-460f-abd0-5af28c9948d5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-pq6d6\" (UID: \"d8807004-fc1f-460f-abd0-5af28c9948d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pq6d6" Jul 08 18:53:23.056767 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:23.056741 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m7wg\" (UniqueName: \"kubernetes.io/projected/d8807004-fc1f-460f-abd0-5af28c9948d5-kube-api-access-8m7wg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-pq6d6\" (UID: \"d8807004-fc1f-460f-abd0-5af28c9948d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pq6d6" Jul 08 18:53:23.146751 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:23.146717 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pq6d6" Jul 08 18:53:23.308067 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:53:23.308031 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8807004_fc1f_460f_abd0_5af28c9948d5.slice/crio-020165c4721ec75202525312469a5242f8dd7d7123ba23ae2ef4730735ae4ed6 WatchSource:0}: Error finding container 020165c4721ec75202525312469a5242f8dd7d7123ba23ae2ef4730735ae4ed6: Status 404 returned error can't find the container with id 020165c4721ec75202525312469a5242f8dd7d7123ba23ae2ef4730735ae4ed6 Jul 08 18:53:23.308175 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:23.308084 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pq6d6"] Jul 08 18:53:23.856907 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:23.856872 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pq6d6" event={"ID":"d8807004-fc1f-460f-abd0-5af28c9948d5","Type":"ContainerStarted","Data":"020165c4721ec75202525312469a5242f8dd7d7123ba23ae2ef4730735ae4ed6"} Jul 08 18:53:25.808625 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:25.808592 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-br2mx" Jul 08 18:53:28.830487 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:28.830454 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qsplg" Jul 08 18:53:30.894803 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:30.894743 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pq6d6" event={"ID":"d8807004-fc1f-460f-abd0-5af28c9948d5","Type":"ContainerStarted","Data":"8d48038ce561b899f57be3588e36e5f933771cd8b1860c4775c52e790a51f2fa"} Jul 08 18:53:30.895306 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:30.894946 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pq6d6" Jul 08 18:53:30.917851 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:30.917799 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pq6d6" podStartSLOduration=1.949909273 podStartE2EDuration="8.917785319s" totalImagesPullingTime="6.967876046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:53:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:53:30.917498328 +0000 UTC m=+471.764772994" watchObservedRunningTime="2026-07-08 18:53:30.917785319 +0000 UTC m=+471.765059984" Jul 08 18:53:31.847543 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:31.847510 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rslwk" Jul 08 18:53:33.214677 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:33.214636 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-nxbdl"] Jul 08 18:53:33.220300 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:33.220279 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nxbdl" Jul 08 18:53:33.223434 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:33.223413 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 08 18:53:33.223744 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:33.223724 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 08 18:53:33.224344 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:33.224325 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8c9hs\"" Jul 08 18:53:33.232715 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:33.232684 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-nxbdl"] Jul 08 18:53:33.339982 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:33.339949 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e076259a-01e3-4020-bae5-6959ef897279-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-nxbdl\" (UID: \"e076259a-01e3-4020-bae5-6959ef897279\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nxbdl" Jul 08 18:53:33.340149 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:33.339989 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2rrx\" (UniqueName: \"kubernetes.io/projected/e076259a-01e3-4020-bae5-6959ef897279-kube-api-access-p2rrx\") pod \"kuadrant-console-plugin-66b5774985-nxbdl\" (UID: \"e076259a-01e3-4020-bae5-6959ef897279\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nxbdl" Jul 08 18:53:33.340149 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:33.340026 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e076259a-01e3-4020-bae5-6959ef897279-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-nxbdl\" (UID: \"e076259a-01e3-4020-bae5-6959ef897279\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nxbdl" Jul 08 18:53:33.441043 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:33.441004 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e076259a-01e3-4020-bae5-6959ef897279-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-nxbdl\" (UID: \"e076259a-01e3-4020-bae5-6959ef897279\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nxbdl" Jul 08 18:53:33.441244 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:33.441105 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e076259a-01e3-4020-bae5-6959ef897279-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-nxbdl\" (UID: \"e076259a-01e3-4020-bae5-6959ef897279\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nxbdl" Jul 08 18:53:33.441244 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:33.441143 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2rrx\" (UniqueName: \"kubernetes.io/projected/e076259a-01e3-4020-bae5-6959ef897279-kube-api-access-p2rrx\") pod \"kuadrant-console-plugin-66b5774985-nxbdl\" (UID: \"e076259a-01e3-4020-bae5-6959ef897279\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nxbdl" Jul 08 18:53:33.441244 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:53:33.441150 2570 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 08 18:53:33.441244 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:53:33.441226 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e076259a-01e3-4020-bae5-6959ef897279-plugin-serving-cert podName:e076259a-01e3-4020-bae5-6959ef897279 nodeName:}" failed. No retries permitted until 2026-07-08 18:53:33.941208035 +0000 UTC m=+474.788482679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/e076259a-01e3-4020-bae5-6959ef897279-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-nxbdl" (UID: "e076259a-01e3-4020-bae5-6959ef897279") : secret "plugin-serving-cert" not found Jul 08 18:53:33.441810 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:33.441792 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e076259a-01e3-4020-bae5-6959ef897279-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-nxbdl\" (UID: \"e076259a-01e3-4020-bae5-6959ef897279\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nxbdl" Jul 08 18:53:33.460228 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:33.460197 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2rrx\" (UniqueName: \"kubernetes.io/projected/e076259a-01e3-4020-bae5-6959ef897279-kube-api-access-p2rrx\") pod \"kuadrant-console-plugin-66b5774985-nxbdl\" (UID: \"e076259a-01e3-4020-bae5-6959ef897279\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nxbdl" Jul 08 18:53:33.945332 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:33.945299 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e076259a-01e3-4020-bae5-6959ef897279-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-nxbdl\" (UID: \"e076259a-01e3-4020-bae5-6959ef897279\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nxbdl" Jul 08 18:53:33.947721 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:33.947688 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e076259a-01e3-4020-bae5-6959ef897279-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-nxbdl\" (UID: \"e076259a-01e3-4020-bae5-6959ef897279\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nxbdl" Jul 08 18:53:34.130848 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:34.130813 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nxbdl" Jul 08 18:53:34.267846 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:34.267811 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-nxbdl"] Jul 08 18:53:34.269521 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:53:34.269488 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode076259a_01e3_4020_bae5_6959ef897279.slice/crio-df934a1f76a88c26fa667f6fae6ebbcb7ee7d97b19e5ede95d47fde2578272bf WatchSource:0}: Error finding container df934a1f76a88c26fa667f6fae6ebbcb7ee7d97b19e5ede95d47fde2578272bf: Status 404 returned error can't find the container with id df934a1f76a88c26fa667f6fae6ebbcb7ee7d97b19e5ede95d47fde2578272bf Jul 08 18:53:34.917497 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:34.917462 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nxbdl" event={"ID":"e076259a-01e3-4020-bae5-6959ef897279","Type":"ContainerStarted","Data":"df934a1f76a88c26fa667f6fae6ebbcb7ee7d97b19e5ede95d47fde2578272bf"} Jul 08 18:53:41.904274 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:41.904224 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pq6d6" Jul 08 18:53:43.321786 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:43.321748 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2lr56"] Jul 08 18:53:43.327931 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:43.327906 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2lr56" Jul 08 18:53:43.342925 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:43.342901 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2lr56"] Jul 08 18:53:43.434846 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:43.434813 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/505cce9a-ad1e-44f6-bd61-429983497a5a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-2lr56\" (UID: \"505cce9a-ad1e-44f6-bd61-429983497a5a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2lr56" Jul 08 18:53:43.434846 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:43.434849 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppfj2\" (UniqueName: \"kubernetes.io/projected/505cce9a-ad1e-44f6-bd61-429983497a5a-kube-api-access-ppfj2\") pod \"kuadrant-operator-controller-manager-55467d9f7-2lr56\" (UID: \"505cce9a-ad1e-44f6-bd61-429983497a5a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2lr56" Jul 08 18:53:43.535538 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:43.535482 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/505cce9a-ad1e-44f6-bd61-429983497a5a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-2lr56\" (UID: \"505cce9a-ad1e-44f6-bd61-429983497a5a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2lr56" Jul 08 18:53:43.535538 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:43.535532 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppfj2\" (UniqueName: \"kubernetes.io/projected/505cce9a-ad1e-44f6-bd61-429983497a5a-kube-api-access-ppfj2\") pod \"kuadrant-operator-controller-manager-55467d9f7-2lr56\" (UID: \"505cce9a-ad1e-44f6-bd61-429983497a5a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2lr56" Jul 08 18:53:43.535938 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:43.535916 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/505cce9a-ad1e-44f6-bd61-429983497a5a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-2lr56\" (UID: \"505cce9a-ad1e-44f6-bd61-429983497a5a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2lr56" Jul 08 18:53:43.547988 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:43.547950 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppfj2\" (UniqueName: \"kubernetes.io/projected/505cce9a-ad1e-44f6-bd61-429983497a5a-kube-api-access-ppfj2\") pod \"kuadrant-operator-controller-manager-55467d9f7-2lr56\" (UID: \"505cce9a-ad1e-44f6-bd61-429983497a5a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2lr56" Jul 08 18:53:43.641396 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:43.641353 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2lr56" Jul 08 18:53:43.804363 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:43.804325 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2lr56"] Jul 08 18:53:43.960500 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:43.960415 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2lr56"] Jul 08 18:53:43.965280 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:43.965235 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2lr56"] Jul 08 18:53:43.992212 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:43.992182 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pq6d6"] Jul 08 18:53:43.992443 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:43.992412 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pq6d6" podUID="d8807004-fc1f-460f-abd0-5af28c9948d5" containerName="manager" containerID="cri-o://8d48038ce561b899f57be3588e36e5f933771cd8b1860c4775c52e790a51f2fa" gracePeriod=2 Jul 08 18:53:44.006386 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:44.006356 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4gvp4"] Jul 08 18:53:44.010676 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:44.010647 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4gvp4" Jul 08 18:53:44.011940 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:44.011909 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pq6d6"] Jul 08 18:53:44.018658 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:53:44.018615 2570 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"d8807004-fc1f-460f-abd0-5af28c9948d5\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-pq6d6\": pods \"kuadrant-operator-controller-manager-8684c9ff49-pq6d6\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pq6d6" Jul 08 18:53:44.024841 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:44.024814 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rslwk"] Jul 08 18:53:44.025081 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:44.025039 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rslwk" podUID="860f6fef-88d8-4642-bbab-d889138c344e" containerName="manager" containerID="cri-o://324cd9d8730afd6f34922dfa02ff93675ff1d8fdc196c6b04ab40ef160dabd0d" gracePeriod=2 Jul 08 18:53:44.045248 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:44.045220 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4gvp4"] Jul 08 18:53:44.048589 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:44.048559 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lvwzm"] Jul 08 18:53:44.050017 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:44.049117 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8807004-fc1f-460f-abd0-5af28c9948d5" containerName="manager" Jul 08 18:53:44.050017 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:44.049137 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8807004-fc1f-460f-abd0-5af28c9948d5" containerName="manager" Jul 08 18:53:44.050017 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:44.049234 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8807004-fc1f-460f-abd0-5af28c9948d5" containerName="manager" Jul 08 18:53:44.052668 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:44.052645 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rslwk"] Jul 08 18:53:44.052783 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:44.052768 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lvwzm" Jul 08 18:53:44.065572 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:53:44.065329 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-pq6d6\" is forbidden: User \"system:node:ip-10-0-130-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-48.ec2.internal' and this object" podUID="d8807004-fc1f-460f-abd0-5af28c9948d5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pq6d6" Jul 08 18:53:44.067066 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:44.067044 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lvwzm"] Jul 08 18:53:44.106663 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:53:44.106629 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-pq6d6\" is forbidden: User \"system:node:ip-10-0-130-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-48.ec2.internal' and this object" podUID="d8807004-fc1f-460f-abd0-5af28c9948d5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pq6d6" Jul 08 18:53:44.109411 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:53:44.109373 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rslwk\" is forbidden: User \"system:node:ip-10-0-130-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-48.ec2.internal' and this object" podUID="860f6fef-88d8-4642-bbab-d889138c344e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rslwk" Jul 08 18:53:44.142873 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:44.142837 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knc6w\" (UniqueName: \"kubernetes.io/projected/8a817375-7339-4598-acfb-0ef08dffe151-kube-api-access-knc6w\") pod \"kuadrant-operator-controller-manager-55467d9f7-4gvp4\" (UID: \"8a817375-7339-4598-acfb-0ef08dffe151\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4gvp4" Jul 08 18:53:44.143023 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:44.142989 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8a817375-7339-4598-acfb-0ef08dffe151-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-4gvp4\" (UID: \"8a817375-7339-4598-acfb-0ef08dffe151\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4gvp4" Jul 08 18:53:44.243984 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:44.243898 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knc6w\" (UniqueName: \"kubernetes.io/projected/8a817375-7339-4598-acfb-0ef08dffe151-kube-api-access-knc6w\") pod \"kuadrant-operator-controller-manager-55467d9f7-4gvp4\" (UID: \"8a817375-7339-4598-acfb-0ef08dffe151\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4gvp4" Jul 08 18:53:44.243984 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:44.243964 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzg8l\" (UniqueName: \"kubernetes.io/projected/c97a96a4-aad6-4930-85e0-575df33d4ca1-kube-api-access-vzg8l\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lvwzm\" (UID: \"c97a96a4-aad6-4930-85e0-575df33d4ca1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lvwzm" Jul 08 18:53:44.244226 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:44.244038 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c97a96a4-aad6-4930-85e0-575df33d4ca1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lvwzm\" (UID: \"c97a96a4-aad6-4930-85e0-575df33d4ca1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lvwzm" Jul 08 18:53:44.244226 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:44.244125 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8a817375-7339-4598-acfb-0ef08dffe151-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-4gvp4\" (UID: \"8a817375-7339-4598-acfb-0ef08dffe151\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4gvp4" Jul 08 18:53:44.244625 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:44.244598 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8a817375-7339-4598-acfb-0ef08dffe151-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-4gvp4\" (UID: \"8a817375-7339-4598-acfb-0ef08dffe151\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4gvp4" Jul 08 18:53:44.255748 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:44.255712 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knc6w\" (UniqueName: \"kubernetes.io/projected/8a817375-7339-4598-acfb-0ef08dffe151-kube-api-access-knc6w\") pod \"kuadrant-operator-controller-manager-55467d9f7-4gvp4\" (UID: \"8a817375-7339-4598-acfb-0ef08dffe151\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4gvp4" Jul 08 18:53:44.344821 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:44.344776 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzg8l\" (UniqueName: \"kubernetes.io/projected/c97a96a4-aad6-4930-85e0-575df33d4ca1-kube-api-access-vzg8l\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lvwzm\" (UID: \"c97a96a4-aad6-4930-85e0-575df33d4ca1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lvwzm" Jul 08 18:53:44.345320 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:44.344866 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c97a96a4-aad6-4930-85e0-575df33d4ca1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lvwzm\" (UID: \"c97a96a4-aad6-4930-85e0-575df33d4ca1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lvwzm" Jul 08 18:53:44.345385 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:44.345344 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c97a96a4-aad6-4930-85e0-575df33d4ca1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lvwzm\" (UID: \"c97a96a4-aad6-4930-85e0-575df33d4ca1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lvwzm" Jul 08 18:53:44.354142 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:44.354118 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzg8l\" (UniqueName: \"kubernetes.io/projected/c97a96a4-aad6-4930-85e0-575df33d4ca1-kube-api-access-vzg8l\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lvwzm\" (UID: \"c97a96a4-aad6-4930-85e0-575df33d4ca1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lvwzm" Jul 08 18:53:44.402583 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:44.402544 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4gvp4" Jul 08 18:53:44.410614 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:44.410463 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lvwzm" Jul 08 18:53:57.853482 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:53:57.853441 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod505cce9a_ad1e_44f6_bd61_429983497a5a.slice/crio-fee6f54cf04c64eb19a7164ba0fafecf1537fa0d7d00ee584859d76cd5ab39e2 WatchSource:0}: Error finding container fee6f54cf04c64eb19a7164ba0fafecf1537fa0d7d00ee584859d76cd5ab39e2: Status 404 returned error can't find the container with id fee6f54cf04c64eb19a7164ba0fafecf1537fa0d7d00ee584859d76cd5ab39e2 Jul 08 18:53:58.030733 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:58.030440 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pq6d6" Jul 08 18:53:58.035396 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:53:58.033524 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-pq6d6\" is forbidden: User \"system:node:ip-10-0-130-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-48.ec2.internal' and this object" podUID="d8807004-fc1f-460f-abd0-5af28c9948d5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pq6d6" Jul 08 18:53:58.055998 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:58.055772 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rslwk" Jul 08 18:53:58.058189 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:53:58.058017 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-pq6d6\" is forbidden: User \"system:node:ip-10-0-130-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-48.ec2.internal' and this object" podUID="d8807004-fc1f-460f-abd0-5af28c9948d5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pq6d6" Jul 08 18:53:58.059655 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:58.059630 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lvwzm"] Jul 08 18:53:58.059921 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:53:58.059893 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rslwk\" is forbidden: User \"system:node:ip-10-0-130-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-48.ec2.internal' and this object" podUID="860f6fef-88d8-4642-bbab-d889138c344e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rslwk" Jul 08 18:53:58.060754 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:53:58.060725 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc97a96a4_aad6_4930_85e0_575df33d4ca1.slice/crio-df82138cb75ed356be83c89c16c95a1be7bf62ee7780be2928dda22b88871701 WatchSource:0}: Error finding container df82138cb75ed356be83c89c16c95a1be7bf62ee7780be2928dda22b88871701: Status 404 returned error can't find the container with id df82138cb75ed356be83c89c16c95a1be7bf62ee7780be2928dda22b88871701 Jul 08 18:53:58.063022 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:58.062880 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2lr56" podUID="505cce9a-ad1e-44f6-bd61-429983497a5a" containerName="manager" containerID="cri-o://cdbac342edbb6b03a3aa2658c9b1eed1f86ae4e932df5108bf9bd74008f355e8" gracePeriod=2 Jul 08 18:53:58.064907 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:58.064875 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nxbdl" event={"ID":"e076259a-01e3-4020-bae5-6959ef897279","Type":"ContainerStarted","Data":"8cad913682e1304e5fa502c5778a24c8fcdce766edbaaf9b2f92169ced38b796"} Jul 08 18:53:58.065468 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:53:58.065431 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-pq6d6\" is forbidden: User \"system:node:ip-10-0-130-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-48.ec2.internal' and this object" podUID="d8807004-fc1f-460f-abd0-5af28c9948d5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pq6d6" Jul 08 18:53:58.066325 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:58.066293 2570 generic.go:356] "Generic (PLEG): container finished" podID="860f6fef-88d8-4642-bbab-d889138c344e" containerID="324cd9d8730afd6f34922dfa02ff93675ff1d8fdc196c6b04ab40ef160dabd0d" exitCode=0 Jul 08 18:53:58.066430 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:58.066329 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rslwk" Jul 08 18:53:58.066430 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:58.066387 2570 scope.go:117] "RemoveContainer" containerID="324cd9d8730afd6f34922dfa02ff93675ff1d8fdc196c6b04ab40ef160dabd0d" Jul 08 18:53:58.067329 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:53:58.067243 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rslwk\" is forbidden: User \"system:node:ip-10-0-130-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-48.ec2.internal' and this object" podUID="860f6fef-88d8-4642-bbab-d889138c344e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rslwk" Jul 08 18:53:58.067657 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:58.067635 2570 generic.go:356] "Generic (PLEG): container finished" podID="d8807004-fc1f-460f-abd0-5af28c9948d5" containerID="8d48038ce561b899f57be3588e36e5f933771cd8b1860c4775c52e790a51f2fa" exitCode=0 Jul 08 18:53:58.067736 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:58.067668 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pq6d6" Jul 08 18:53:58.069200 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:53:58.069179 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-2lr56\" is forbidden: User \"system:node:ip-10-0-130-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-48.ec2.internal' and this object" podUID="505cce9a-ad1e-44f6-bd61-429983497a5a" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2lr56" Jul 08 18:53:58.071010 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:53:58.070986 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-2lr56\" is forbidden: User \"system:node:ip-10-0-130-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-48.ec2.internal' and this object" podUID="505cce9a-ad1e-44f6-bd61-429983497a5a" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2lr56" Jul 08 18:53:58.072835 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:53:58.072813 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-pq6d6\" is forbidden: User \"system:node:ip-10-0-130-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-48.ec2.internal' and this object" podUID="d8807004-fc1f-460f-abd0-5af28c9948d5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pq6d6" Jul 08 18:53:58.074654 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:53:58.074632 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rslwk\" is forbidden: User \"system:node:ip-10-0-130-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-48.ec2.internal' and this object" podUID="860f6fef-88d8-4642-bbab-d889138c344e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rslwk" Jul 08 18:53:58.075871 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:58.075857 2570 scope.go:117] "RemoveContainer" containerID="324cd9d8730afd6f34922dfa02ff93675ff1d8fdc196c6b04ab40ef160dabd0d" Jul 08 18:53:58.076120 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:53:58.076102 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"324cd9d8730afd6f34922dfa02ff93675ff1d8fdc196c6b04ab40ef160dabd0d\": container with ID starting with 324cd9d8730afd6f34922dfa02ff93675ff1d8fdc196c6b04ab40ef160dabd0d not found: ID does not exist" containerID="324cd9d8730afd6f34922dfa02ff93675ff1d8fdc196c6b04ab40ef160dabd0d" Jul 08 18:53:58.076174 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:58.076125 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"324cd9d8730afd6f34922dfa02ff93675ff1d8fdc196c6b04ab40ef160dabd0d"} err="failed to get container status \"324cd9d8730afd6f34922dfa02ff93675ff1d8fdc196c6b04ab40ef160dabd0d\": rpc error: code = NotFound desc = could not find container \"324cd9d8730afd6f34922dfa02ff93675ff1d8fdc196c6b04ab40ef160dabd0d\": container with ID starting with 324cd9d8730afd6f34922dfa02ff93675ff1d8fdc196c6b04ab40ef160dabd0d not found: ID does not exist" Jul 08 18:53:58.076174 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:58.076140 2570 scope.go:117] "RemoveContainer" containerID="8d48038ce561b899f57be3588e36e5f933771cd8b1860c4775c52e790a51f2fa" Jul 08 18:53:58.077104 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:58.077086 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8m7wg\" (UniqueName: \"kubernetes.io/projected/d8807004-fc1f-460f-abd0-5af28c9948d5-kube-api-access-8m7wg\") pod \"d8807004-fc1f-460f-abd0-5af28c9948d5\" (UID: \"d8807004-fc1f-460f-abd0-5af28c9948d5\") " Jul 08 18:53:58.077177 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:58.077160 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8807004-fc1f-460f-abd0-5af28c9948d5-extensions-socket-volume\") pod \"d8807004-fc1f-460f-abd0-5af28c9948d5\" (UID: \"d8807004-fc1f-460f-abd0-5af28c9948d5\") " Jul 08 18:53:58.077558 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:58.077533 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8807004-fc1f-460f-abd0-5af28c9948d5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d8807004-fc1f-460f-abd0-5af28c9948d5" (UID: "d8807004-fc1f-460f-abd0-5af28c9948d5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:53:58.079165 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:58.079135 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8807004-fc1f-460f-abd0-5af28c9948d5-kube-api-access-8m7wg" (OuterVolumeSpecName: "kube-api-access-8m7wg") pod "d8807004-fc1f-460f-abd0-5af28c9948d5" (UID: "d8807004-fc1f-460f-abd0-5af28c9948d5"). InnerVolumeSpecName "kube-api-access-8m7wg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:53:58.084785 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:58.084759 2570 scope.go:117] "RemoveContainer" containerID="8d48038ce561b899f57be3588e36e5f933771cd8b1860c4775c52e790a51f2fa" Jul 08 18:53:58.085148 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:53:58.085117 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d48038ce561b899f57be3588e36e5f933771cd8b1860c4775c52e790a51f2fa\": container with ID starting with 8d48038ce561b899f57be3588e36e5f933771cd8b1860c4775c52e790a51f2fa not found: ID does not exist" containerID="8d48038ce561b899f57be3588e36e5f933771cd8b1860c4775c52e790a51f2fa" Jul 08 18:53:58.085148 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:58.085142 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d48038ce561b899f57be3588e36e5f933771cd8b1860c4775c52e790a51f2fa"} err="failed to get container status \"8d48038ce561b899f57be3588e36e5f933771cd8b1860c4775c52e790a51f2fa\": rpc error: code = NotFound desc = could not find container \"8d48038ce561b899f57be3588e36e5f933771cd8b1860c4775c52e790a51f2fa\": container with ID starting with 8d48038ce561b899f57be3588e36e5f933771cd8b1860c4775c52e790a51f2fa not found: ID does not exist" Jul 08 18:53:58.087369 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:58.087349 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4gvp4"] Jul 08 18:53:58.094112 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:58.094074 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nxbdl" podStartSLOduration=1.431962108 podStartE2EDuration="25.094062737s" totalImagesPullingTime="23.662100629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:53:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:53:58.092860288 +0000 UTC m=+498.940134956" watchObservedRunningTime="2026-07-08 18:53:58.094062737 +0000 UTC m=+498.941337403" Jul 08 18:53:58.095354 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:53:58.095328 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a817375_7339_4598_acfb_0ef08dffe151.slice/crio-3a74b3d7be30402b2d9523ecb9c8b41787267538f7a592bb1f9ffc9384409443 WatchSource:0}: Error finding container 3a74b3d7be30402b2d9523ecb9c8b41787267538f7a592bb1f9ffc9384409443: Status 404 returned error can't find the container with id 3a74b3d7be30402b2d9523ecb9c8b41787267538f7a592bb1f9ffc9384409443 Jul 08 18:53:58.178208 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:58.178178 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8pr9\" (UniqueName: \"kubernetes.io/projected/860f6fef-88d8-4642-bbab-d889138c344e-kube-api-access-t8pr9\") pod \"860f6fef-88d8-4642-bbab-d889138c344e\" (UID: \"860f6fef-88d8-4642-bbab-d889138c344e\") " Jul 08 18:53:58.178504 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:58.178484 2570 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8807004-fc1f-460f-abd0-5af28c9948d5-extensions-socket-volume\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:53:58.178587 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:58.178509 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m7wg\" (UniqueName: \"kubernetes.io/projected/d8807004-fc1f-460f-abd0-5af28c9948d5-kube-api-access-8m7wg\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:53:58.180105 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:58.180074 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/860f6fef-88d8-4642-bbab-d889138c344e-kube-api-access-t8pr9" (OuterVolumeSpecName: "kube-api-access-t8pr9") pod "860f6fef-88d8-4642-bbab-d889138c344e" (UID: "860f6fef-88d8-4642-bbab-d889138c344e"). InnerVolumeSpecName "kube-api-access-t8pr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:53:58.279921 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:58.279892 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8pr9\" (UniqueName: \"kubernetes.io/projected/860f6fef-88d8-4642-bbab-d889138c344e-kube-api-access-t8pr9\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:53:58.291579 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:58.291557 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2lr56" Jul 08 18:53:58.294138 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:53:58.294116 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rslwk\" is forbidden: User \"system:node:ip-10-0-130-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-48.ec2.internal' and this object" podUID="860f6fef-88d8-4642-bbab-d889138c344e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rslwk" Jul 08 18:53:58.296136 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:53:58.296115 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-2lr56\" is forbidden: User \"system:node:ip-10-0-130-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-48.ec2.internal' and this object" podUID="505cce9a-ad1e-44f6-bd61-429983497a5a" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2lr56" Jul 08 18:53:58.298097 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:53:58.298081 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-pq6d6\" is forbidden: User \"system:node:ip-10-0-130-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-48.ec2.internal' and this object" podUID="d8807004-fc1f-460f-abd0-5af28c9948d5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pq6d6" Jul 08 18:53:58.379603 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:53:58.379573 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-2lr56\" is forbidden: User \"system:node:ip-10-0-130-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-48.ec2.internal' and this object" podUID="505cce9a-ad1e-44f6-bd61-429983497a5a" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2lr56" Jul 08 18:53:58.380807 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:58.380789 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/505cce9a-ad1e-44f6-bd61-429983497a5a-extensions-socket-volume\") pod \"505cce9a-ad1e-44f6-bd61-429983497a5a\" (UID: \"505cce9a-ad1e-44f6-bd61-429983497a5a\") " Jul 08 18:53:58.380876 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:58.380829 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppfj2\" (UniqueName: \"kubernetes.io/projected/505cce9a-ad1e-44f6-bd61-429983497a5a-kube-api-access-ppfj2\") pod \"505cce9a-ad1e-44f6-bd61-429983497a5a\" (UID: \"505cce9a-ad1e-44f6-bd61-429983497a5a\") " Jul 08 18:53:58.381020 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:58.381001 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/505cce9a-ad1e-44f6-bd61-429983497a5a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "505cce9a-ad1e-44f6-bd61-429983497a5a" (UID: "505cce9a-ad1e-44f6-bd61-429983497a5a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:53:58.381609 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:53:58.381589 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-pq6d6\" is forbidden: User \"system:node:ip-10-0-130-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-48.ec2.internal' and this object" podUID="d8807004-fc1f-460f-abd0-5af28c9948d5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pq6d6" Jul 08 18:53:58.382723 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:58.382705 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/505cce9a-ad1e-44f6-bd61-429983497a5a-kube-api-access-ppfj2" (OuterVolumeSpecName: "kube-api-access-ppfj2") pod "505cce9a-ad1e-44f6-bd61-429983497a5a" (UID: "505cce9a-ad1e-44f6-bd61-429983497a5a"). InnerVolumeSpecName "kube-api-access-ppfj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:53:58.383598 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:53:58.383579 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rslwk\" is forbidden: User \"system:node:ip-10-0-130-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-48.ec2.internal' and this object" podUID="860f6fef-88d8-4642-bbab-d889138c344e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rslwk" Jul 08 18:53:58.385701 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:53:58.385676 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-2lr56\" is forbidden: User \"system:node:ip-10-0-130-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-48.ec2.internal' and this object" podUID="505cce9a-ad1e-44f6-bd61-429983497a5a" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2lr56" Jul 08 18:53:58.387729 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:53:58.387709 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-pq6d6\" is forbidden: User \"system:node:ip-10-0-130-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-48.ec2.internal' and this object" podUID="d8807004-fc1f-460f-abd0-5af28c9948d5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pq6d6" Jul 08 18:53:58.390422 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:53:58.390401 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rslwk\" is forbidden: User \"system:node:ip-10-0-130-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-48.ec2.internal' and this object" podUID="860f6fef-88d8-4642-bbab-d889138c344e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rslwk" Jul 08 18:53:58.482318 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:58.482288 2570 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/505cce9a-ad1e-44f6-bd61-429983497a5a-extensions-socket-volume\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:53:58.482318 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:58.482314 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppfj2\" (UniqueName: \"kubernetes.io/projected/505cce9a-ad1e-44f6-bd61-429983497a5a-kube-api-access-ppfj2\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:53:59.075813 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:59.075778 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4gvp4" event={"ID":"8a817375-7339-4598-acfb-0ef08dffe151","Type":"ContainerStarted","Data":"40358f5075ba8af86ce23804f1fe1217964ee0eae4859f364607e3fa5c18c133"} Jul 08 18:53:59.076202 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:59.075828 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4gvp4" event={"ID":"8a817375-7339-4598-acfb-0ef08dffe151","Type":"ContainerStarted","Data":"3a74b3d7be30402b2d9523ecb9c8b41787267538f7a592bb1f9ffc9384409443"} Jul 08 18:53:59.076202 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:59.075885 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4gvp4" Jul 08 18:53:59.077987 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:59.077966 2570 generic.go:356] "Generic (PLEG): container finished" podID="505cce9a-ad1e-44f6-bd61-429983497a5a" containerID="cdbac342edbb6b03a3aa2658c9b1eed1f86ae4e932df5108bf9bd74008f355e8" exitCode=2 Jul 08 18:53:59.078117 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:59.078000 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2lr56" Jul 08 18:53:59.078117 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:59.078051 2570 scope.go:117] "RemoveContainer" containerID="cdbac342edbb6b03a3aa2658c9b1eed1f86ae4e932df5108bf9bd74008f355e8" Jul 08 18:53:59.078402 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:53:59.078378 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-2lr56\" is forbidden: User \"system:node:ip-10-0-130-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-48.ec2.internal' and this object" podUID="505cce9a-ad1e-44f6-bd61-429983497a5a" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2lr56" Jul 08 18:53:59.079800 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:59.079778 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lvwzm" event={"ID":"c97a96a4-aad6-4930-85e0-575df33d4ca1","Type":"ContainerStarted","Data":"6bfbbd5f2b877cafd06f78f6a4f66d449c81af768ca3841efd3303ba09e40e73"} Jul 08 18:53:59.079909 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:59.079835 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lvwzm" event={"ID":"c97a96a4-aad6-4930-85e0-575df33d4ca1","Type":"ContainerStarted","Data":"df82138cb75ed356be83c89c16c95a1be7bf62ee7780be2928dda22b88871701"} Jul 08 18:53:59.089681 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:59.089662 2570 scope.go:117] "RemoveContainer" containerID="cdbac342edbb6b03a3aa2658c9b1eed1f86ae4e932df5108bf9bd74008f355e8" Jul 08 18:53:59.090275 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:53:59.090218 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdbac342edbb6b03a3aa2658c9b1eed1f86ae4e932df5108bf9bd74008f355e8\": container with ID starting with cdbac342edbb6b03a3aa2658c9b1eed1f86ae4e932df5108bf9bd74008f355e8 not found: ID does not exist" containerID="cdbac342edbb6b03a3aa2658c9b1eed1f86ae4e932df5108bf9bd74008f355e8" Jul 08 18:53:59.090395 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:59.090309 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdbac342edbb6b03a3aa2658c9b1eed1f86ae4e932df5108bf9bd74008f355e8"} err="failed to get container status \"cdbac342edbb6b03a3aa2658c9b1eed1f86ae4e932df5108bf9bd74008f355e8\": rpc error: code = NotFound desc = could not find container \"cdbac342edbb6b03a3aa2658c9b1eed1f86ae4e932df5108bf9bd74008f355e8\": container with ID starting with cdbac342edbb6b03a3aa2658c9b1eed1f86ae4e932df5108bf9bd74008f355e8 not found: ID does not exist" Jul 08 18:53:59.102578 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:53:59.102549 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-pq6d6\" is forbidden: User \"system:node:ip-10-0-130-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-48.ec2.internal' and this object" podUID="d8807004-fc1f-460f-abd0-5af28c9948d5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pq6d6" Jul 08 18:53:59.103970 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:59.103934 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4gvp4" podStartSLOduration=16.103923519 podStartE2EDuration="16.103923519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:53:59.099998169 +0000 UTC m=+499.947272834" watchObservedRunningTime="2026-07-08 18:53:59.103923519 +0000 UTC m=+499.951198184" Jul 08 18:53:59.108955 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:53:59.108903 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rslwk\" is forbidden: User \"system:node:ip-10-0-130-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-48.ec2.internal' and this object" podUID="860f6fef-88d8-4642-bbab-d889138c344e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rslwk" Jul 08 18:53:59.113975 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:53:59.113953 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-2lr56\" is forbidden: User \"system:node:ip-10-0-130-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-48.ec2.internal' and this object" podUID="505cce9a-ad1e-44f6-bd61-429983497a5a" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2lr56" Jul 08 18:53:59.140150 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:53:59.140124 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-pq6d6\" is forbidden: User \"system:node:ip-10-0-130-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-48.ec2.internal' and this object" podUID="d8807004-fc1f-460f-abd0-5af28c9948d5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pq6d6" Jul 08 18:53:59.140746 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:59.140709 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lvwzm" podStartSLOduration=15.14069653 podStartE2EDuration="15.14069653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:53:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:53:59.138045748 +0000 UTC m=+499.985320417" watchObservedRunningTime="2026-07-08 18:53:59.14069653 +0000 UTC m=+499.987971195" Jul 08 18:53:59.142914 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:53:59.142893 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rslwk\" is forbidden: User \"system:node:ip-10-0-130-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-48.ec2.internal' and this object" podUID="860f6fef-88d8-4642-bbab-d889138c344e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rslwk" Jul 08 18:53:59.688411 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:53:59.688374 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-pq6d6\" is forbidden: User \"system:node:ip-10-0-130-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-48.ec2.internal' and this object" podUID="d8807004-fc1f-460f-abd0-5af28c9948d5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pq6d6" Jul 08 18:53:59.689857 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:59.689836 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="505cce9a-ad1e-44f6-bd61-429983497a5a" path="/var/lib/kubelet/pods/505cce9a-ad1e-44f6-bd61-429983497a5a/volumes" Jul 08 18:53:59.690154 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:59.690142 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="860f6fef-88d8-4642-bbab-d889138c344e" path="/var/lib/kubelet/pods/860f6fef-88d8-4642-bbab-d889138c344e/volumes" Jul 08 18:53:59.690490 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:53:59.690478 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8807004-fc1f-460f-abd0-5af28c9948d5" path="/var/lib/kubelet/pods/d8807004-fc1f-460f-abd0-5af28c9948d5/volumes" Jul 08 18:53:59.690753 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:53:59.690731 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rslwk\" is forbidden: User \"system:node:ip-10-0-130-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-48.ec2.internal' and this object" podUID="860f6fef-88d8-4642-bbab-d889138c344e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rslwk" Jul 08 18:53:59.692871 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:53:59.692852 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-2lr56\" is forbidden: User \"system:node:ip-10-0-130-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-48.ec2.internal' and this object" podUID="505cce9a-ad1e-44f6-bd61-429983497a5a" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2lr56" Jul 08 18:54:00.086720 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:00.086642 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lvwzm" Jul 08 18:54:10.089313 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:10.089280 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4gvp4" Jul 08 18:54:10.145816 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:10.145782 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lvwzm"] Jul 08 18:54:10.146108 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:10.146046 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lvwzm" podUID="c97a96a4-aad6-4930-85e0-575df33d4ca1" containerName="manager" containerID="cri-o://6bfbbd5f2b877cafd06f78f6a4f66d449c81af768ca3841efd3303ba09e40e73" gracePeriod=10 Jul 08 18:54:10.147799 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:10.147778 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lvwzm" Jul 08 18:54:10.399310 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:10.399279 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lvwzm" Jul 08 18:54:10.445268 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:10.445218 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tkzsc"] Jul 08 18:54:10.445618 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:10.445602 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="505cce9a-ad1e-44f6-bd61-429983497a5a" containerName="manager" Jul 08 18:54:10.445712 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:10.445620 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="505cce9a-ad1e-44f6-bd61-429983497a5a" containerName="manager" Jul 08 18:54:10.445712 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:10.445635 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c97a96a4-aad6-4930-85e0-575df33d4ca1" containerName="manager" Jul 08 18:54:10.445712 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:10.445641 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c97a96a4-aad6-4930-85e0-575df33d4ca1" containerName="manager" Jul 08 18:54:10.445712 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:10.445650 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="860f6fef-88d8-4642-bbab-d889138c344e" containerName="manager" Jul 08 18:54:10.445712 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:10.445655 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="860f6fef-88d8-4642-bbab-d889138c344e" containerName="manager" Jul 08 18:54:10.445987 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:10.445733 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="505cce9a-ad1e-44f6-bd61-429983497a5a" containerName="manager" Jul 08 18:54:10.445987 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:10.445741 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="860f6fef-88d8-4642-bbab-d889138c344e" containerName="manager" Jul 08 18:54:10.445987 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:10.445747 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c97a96a4-aad6-4930-85e0-575df33d4ca1" containerName="manager" Jul 08 18:54:10.447858 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:10.447818 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tkzsc" Jul 08 18:54:10.464877 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:10.464855 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tkzsc"] Jul 08 18:54:10.481321 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:10.481300 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzg8l\" (UniqueName: \"kubernetes.io/projected/c97a96a4-aad6-4930-85e0-575df33d4ca1-kube-api-access-vzg8l\") pod \"c97a96a4-aad6-4930-85e0-575df33d4ca1\" (UID: \"c97a96a4-aad6-4930-85e0-575df33d4ca1\") " Jul 08 18:54:10.481429 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:10.481399 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c97a96a4-aad6-4930-85e0-575df33d4ca1-extensions-socket-volume\") pod \"c97a96a4-aad6-4930-85e0-575df33d4ca1\" (UID: \"c97a96a4-aad6-4930-85e0-575df33d4ca1\") " Jul 08 18:54:10.481751 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:10.481729 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c97a96a4-aad6-4930-85e0-575df33d4ca1-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c97a96a4-aad6-4930-85e0-575df33d4ca1" (UID: "c97a96a4-aad6-4930-85e0-575df33d4ca1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:54:10.483132 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:10.483113 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c97a96a4-aad6-4930-85e0-575df33d4ca1-kube-api-access-vzg8l" (OuterVolumeSpecName: "kube-api-access-vzg8l") pod "c97a96a4-aad6-4930-85e0-575df33d4ca1" (UID: "c97a96a4-aad6-4930-85e0-575df33d4ca1"). InnerVolumeSpecName "kube-api-access-vzg8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:54:10.581974 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:10.581944 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/832b207d-e75f-40b9-89b4-79421084796a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-tkzsc\" (UID: \"832b207d-e75f-40b9-89b4-79421084796a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tkzsc" Jul 08 18:54:10.582109 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:10.581992 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg876\" (UniqueName: \"kubernetes.io/projected/832b207d-e75f-40b9-89b4-79421084796a-kube-api-access-kg876\") pod \"kuadrant-operator-controller-manager-86878c769f-tkzsc\" (UID: \"832b207d-e75f-40b9-89b4-79421084796a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tkzsc" Jul 08 18:54:10.582109 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:10.582035 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzg8l\" (UniqueName: \"kubernetes.io/projected/c97a96a4-aad6-4930-85e0-575df33d4ca1-kube-api-access-vzg8l\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:54:10.582109 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:10.582045 2570 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c97a96a4-aad6-4930-85e0-575df33d4ca1-extensions-socket-volume\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:54:10.683460 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:10.683372 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/832b207d-e75f-40b9-89b4-79421084796a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-tkzsc\" (UID: \"832b207d-e75f-40b9-89b4-79421084796a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tkzsc" Jul 08 18:54:10.683460 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:10.683440 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg876\" (UniqueName: \"kubernetes.io/projected/832b207d-e75f-40b9-89b4-79421084796a-kube-api-access-kg876\") pod \"kuadrant-operator-controller-manager-86878c769f-tkzsc\" (UID: \"832b207d-e75f-40b9-89b4-79421084796a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tkzsc" Jul 08 18:54:10.683727 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:10.683706 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/832b207d-e75f-40b9-89b4-79421084796a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-tkzsc\" (UID: \"832b207d-e75f-40b9-89b4-79421084796a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tkzsc" Jul 08 18:54:10.695927 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:10.695901 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg876\" (UniqueName: \"kubernetes.io/projected/832b207d-e75f-40b9-89b4-79421084796a-kube-api-access-kg876\") pod \"kuadrant-operator-controller-manager-86878c769f-tkzsc\" (UID: \"832b207d-e75f-40b9-89b4-79421084796a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tkzsc" Jul 08 18:54:10.758496 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:10.758467 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tkzsc" Jul 08 18:54:10.908962 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:10.908937 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tkzsc"] Jul 08 18:54:10.910072 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:54:10.910048 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod832b207d_e75f_40b9_89b4_79421084796a.slice/crio-5ed79674a746f063099f7b6177d69df4052354b8fec070b77b9ef7cfc9df5b16 WatchSource:0}: Error finding container 5ed79674a746f063099f7b6177d69df4052354b8fec070b77b9ef7cfc9df5b16: Status 404 returned error can't find the container with id 5ed79674a746f063099f7b6177d69df4052354b8fec070b77b9ef7cfc9df5b16 Jul 08 18:54:11.145810 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:11.145774 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tkzsc" event={"ID":"832b207d-e75f-40b9-89b4-79421084796a","Type":"ContainerStarted","Data":"fa77a10a865efbac98000125c462f764c9be787379e01718b26117ce6c3b1563"} Jul 08 18:54:11.145810 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:11.145844 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tkzsc" Jul 08 18:54:11.146406 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:11.145871 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tkzsc" event={"ID":"832b207d-e75f-40b9-89b4-79421084796a","Type":"ContainerStarted","Data":"5ed79674a746f063099f7b6177d69df4052354b8fec070b77b9ef7cfc9df5b16"} Jul 08 18:54:11.147671 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:11.147644 2570 generic.go:356] "Generic (PLEG): container finished" podID="c97a96a4-aad6-4930-85e0-575df33d4ca1" containerID="6bfbbd5f2b877cafd06f78f6a4f66d449c81af768ca3841efd3303ba09e40e73" exitCode=0 Jul 08 18:54:11.147789 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:11.147692 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lvwzm" Jul 08 18:54:11.147789 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:11.147703 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lvwzm" event={"ID":"c97a96a4-aad6-4930-85e0-575df33d4ca1","Type":"ContainerDied","Data":"6bfbbd5f2b877cafd06f78f6a4f66d449c81af768ca3841efd3303ba09e40e73"} Jul 08 18:54:11.147789 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:11.147761 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lvwzm" event={"ID":"c97a96a4-aad6-4930-85e0-575df33d4ca1","Type":"ContainerDied","Data":"df82138cb75ed356be83c89c16c95a1be7bf62ee7780be2928dda22b88871701"} Jul 08 18:54:11.147906 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:11.147796 2570 scope.go:117] "RemoveContainer" containerID="6bfbbd5f2b877cafd06f78f6a4f66d449c81af768ca3841efd3303ba09e40e73" Jul 08 18:54:11.158426 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:11.158413 2570 scope.go:117] "RemoveContainer" containerID="6bfbbd5f2b877cafd06f78f6a4f66d449c81af768ca3841efd3303ba09e40e73" Jul 08 18:54:11.158665 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:54:11.158646 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bfbbd5f2b877cafd06f78f6a4f66d449c81af768ca3841efd3303ba09e40e73\": container with ID starting with 6bfbbd5f2b877cafd06f78f6a4f66d449c81af768ca3841efd3303ba09e40e73 not found: ID does not exist" containerID="6bfbbd5f2b877cafd06f78f6a4f66d449c81af768ca3841efd3303ba09e40e73" Jul 08 18:54:11.158727 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:11.158680 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bfbbd5f2b877cafd06f78f6a4f66d449c81af768ca3841efd3303ba09e40e73"} err="failed to get container status \"6bfbbd5f2b877cafd06f78f6a4f66d449c81af768ca3841efd3303ba09e40e73\": rpc error: code = NotFound desc = could not find container \"6bfbbd5f2b877cafd06f78f6a4f66d449c81af768ca3841efd3303ba09e40e73\": container with ID starting with 6bfbbd5f2b877cafd06f78f6a4f66d449c81af768ca3841efd3303ba09e40e73 not found: ID does not exist" Jul 08 18:54:11.175245 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:11.175206 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tkzsc" podStartSLOduration=1.175192785 podStartE2EDuration="1.175192785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:54:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:54:11.173022402 +0000 UTC m=+512.020297068" watchObservedRunningTime="2026-07-08 18:54:11.175192785 +0000 UTC m=+512.022467451" Jul 08 18:54:11.194818 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:11.194797 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lvwzm"] Jul 08 18:54:11.201675 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:11.201657 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lvwzm"] Jul 08 18:54:11.690428 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:11.690399 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c97a96a4-aad6-4930-85e0-575df33d4ca1" path="/var/lib/kubelet/pods/c97a96a4-aad6-4930-85e0-575df33d4ca1/volumes" Jul 08 18:54:22.157100 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:22.157063 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tkzsc" Jul 08 18:54:22.210623 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:22.210588 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4gvp4"] Jul 08 18:54:22.211238 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:22.211201 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4gvp4" podUID="8a817375-7339-4598-acfb-0ef08dffe151" containerName="manager" containerID="cri-o://40358f5075ba8af86ce23804f1fe1217964ee0eae4859f364607e3fa5c18c133" gracePeriod=10 Jul 08 18:54:22.977215 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:22.977190 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4gvp4" Jul 08 18:54:23.089989 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:23.089899 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knc6w\" (UniqueName: \"kubernetes.io/projected/8a817375-7339-4598-acfb-0ef08dffe151-kube-api-access-knc6w\") pod \"8a817375-7339-4598-acfb-0ef08dffe151\" (UID: \"8a817375-7339-4598-acfb-0ef08dffe151\") " Jul 08 18:54:23.089989 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:23.089959 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8a817375-7339-4598-acfb-0ef08dffe151-extensions-socket-volume\") pod \"8a817375-7339-4598-acfb-0ef08dffe151\" (UID: \"8a817375-7339-4598-acfb-0ef08dffe151\") " Jul 08 18:54:23.090417 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:23.090377 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a817375-7339-4598-acfb-0ef08dffe151-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8a817375-7339-4598-acfb-0ef08dffe151" (UID: "8a817375-7339-4598-acfb-0ef08dffe151"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:54:23.092042 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:23.092017 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a817375-7339-4598-acfb-0ef08dffe151-kube-api-access-knc6w" (OuterVolumeSpecName: "kube-api-access-knc6w") pod "8a817375-7339-4598-acfb-0ef08dffe151" (UID: "8a817375-7339-4598-acfb-0ef08dffe151"). InnerVolumeSpecName "kube-api-access-knc6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:54:23.191178 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:23.191148 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knc6w\" (UniqueName: \"kubernetes.io/projected/8a817375-7339-4598-acfb-0ef08dffe151-kube-api-access-knc6w\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:54:23.191178 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:23.191171 2570 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8a817375-7339-4598-acfb-0ef08dffe151-extensions-socket-volume\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:54:23.221030 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:23.220995 2570 generic.go:356] "Generic (PLEG): container finished" podID="8a817375-7339-4598-acfb-0ef08dffe151" containerID="40358f5075ba8af86ce23804f1fe1217964ee0eae4859f364607e3fa5c18c133" exitCode=0 Jul 08 18:54:23.221139 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:23.221037 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4gvp4" Jul 08 18:54:23.221139 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:23.221072 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4gvp4" event={"ID":"8a817375-7339-4598-acfb-0ef08dffe151","Type":"ContainerDied","Data":"40358f5075ba8af86ce23804f1fe1217964ee0eae4859f364607e3fa5c18c133"} Jul 08 18:54:23.221139 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:23.221122 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4gvp4" event={"ID":"8a817375-7339-4598-acfb-0ef08dffe151","Type":"ContainerDied","Data":"3a74b3d7be30402b2d9523ecb9c8b41787267538f7a592bb1f9ffc9384409443"} Jul 08 18:54:23.221247 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:23.221146 2570 scope.go:117] "RemoveContainer" containerID="40358f5075ba8af86ce23804f1fe1217964ee0eae4859f364607e3fa5c18c133" Jul 08 18:54:23.231118 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:23.231102 2570 scope.go:117] "RemoveContainer" containerID="40358f5075ba8af86ce23804f1fe1217964ee0eae4859f364607e3fa5c18c133" Jul 08 18:54:23.231420 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:54:23.231402 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40358f5075ba8af86ce23804f1fe1217964ee0eae4859f364607e3fa5c18c133\": container with ID starting with 40358f5075ba8af86ce23804f1fe1217964ee0eae4859f364607e3fa5c18c133 not found: ID does not exist" containerID="40358f5075ba8af86ce23804f1fe1217964ee0eae4859f364607e3fa5c18c133" Jul 08 18:54:23.231492 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:23.231430 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40358f5075ba8af86ce23804f1fe1217964ee0eae4859f364607e3fa5c18c133"} err="failed to get container status \"40358f5075ba8af86ce23804f1fe1217964ee0eae4859f364607e3fa5c18c133\": rpc error: code = NotFound desc = could not find container \"40358f5075ba8af86ce23804f1fe1217964ee0eae4859f364607e3fa5c18c133\": container with ID starting with 40358f5075ba8af86ce23804f1fe1217964ee0eae4859f364607e3fa5c18c133 not found: ID does not exist" Jul 08 18:54:23.266982 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:23.266960 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4gvp4"] Jul 08 18:54:23.283234 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:23.283212 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-4gvp4"] Jul 08 18:54:23.690064 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:23.690031 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a817375-7339-4598-acfb-0ef08dffe151" path="/var/lib/kubelet/pods/8a817375-7339-4598-acfb-0ef08dffe151/volumes" Jul 08 18:54:26.552715 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:26.552682 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8"] Jul 08 18:54:26.553180 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:26.553049 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a817375-7339-4598-acfb-0ef08dffe151" containerName="manager" Jul 08 18:54:26.553180 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:26.553060 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a817375-7339-4598-acfb-0ef08dffe151" containerName="manager" Jul 08 18:54:26.553180 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:26.553133 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a817375-7339-4598-acfb-0ef08dffe151" containerName="manager" Jul 08 18:54:26.578187 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:26.578154 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8"] Jul 08 18:54:26.578418 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:26.578305 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:54:26.581587 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:26.581568 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-ptjcw\"" Jul 08 18:54:26.581737 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:26.581572 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 08 18:54:26.722842 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:26.722805 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a2a709-efb7-4820-831e-178d5bfb30c8-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-g96h8\" (UID: \"d8a2a709-efb7-4820-831e-178d5bfb30c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:54:26.722842 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:26.722844 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a2a709-efb7-4820-831e-178d5bfb30c8-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-g96h8\" (UID: \"d8a2a709-efb7-4820-831e-178d5bfb30c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:54:26.723061 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:26.722866 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a2a709-efb7-4820-831e-178d5bfb30c8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-g96h8\" (UID: \"d8a2a709-efb7-4820-831e-178d5bfb30c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:54:26.723061 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:26.722895 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8a2a709-efb7-4820-831e-178d5bfb30c8-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-g96h8\" (UID: \"d8a2a709-efb7-4820-831e-178d5bfb30c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:54:26.723061 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:26.722921 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4s9h\" (UniqueName: \"kubernetes.io/projected/d8a2a709-efb7-4820-831e-178d5bfb30c8-kube-api-access-h4s9h\") pod \"maas-default-gateway-openshift-default-56654dcf7-g96h8\" (UID: \"d8a2a709-efb7-4820-831e-178d5bfb30c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:54:26.723061 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:26.722943 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a2a709-efb7-4820-831e-178d5bfb30c8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-g96h8\" (UID: \"d8a2a709-efb7-4820-831e-178d5bfb30c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:54:26.723061 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:26.722965 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a2a709-efb7-4820-831e-178d5bfb30c8-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-g96h8\" (UID: \"d8a2a709-efb7-4820-831e-178d5bfb30c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:54:26.723061 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:26.723035 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a2a709-efb7-4820-831e-178d5bfb30c8-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-g96h8\" (UID: \"d8a2a709-efb7-4820-831e-178d5bfb30c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:54:26.723061 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:26.723059 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a2a709-efb7-4820-831e-178d5bfb30c8-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-g96h8\" (UID: \"d8a2a709-efb7-4820-831e-178d5bfb30c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:54:26.823833 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:26.823763 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a2a709-efb7-4820-831e-178d5bfb30c8-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-g96h8\" (UID: \"d8a2a709-efb7-4820-831e-178d5bfb30c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:54:26.823833 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:26.823796 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a2a709-efb7-4820-831e-178d5bfb30c8-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-g96h8\" (UID: \"d8a2a709-efb7-4820-831e-178d5bfb30c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:54:26.823833 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:26.823818 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a2a709-efb7-4820-831e-178d5bfb30c8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-g96h8\" (UID: \"d8a2a709-efb7-4820-831e-178d5bfb30c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:54:26.824087 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:26.824015 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8a2a709-efb7-4820-831e-178d5bfb30c8-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-g96h8\" (UID: \"d8a2a709-efb7-4820-831e-178d5bfb30c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:54:26.824087 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:26.824076 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4s9h\" (UniqueName: \"kubernetes.io/projected/d8a2a709-efb7-4820-831e-178d5bfb30c8-kube-api-access-h4s9h\") pod \"maas-default-gateway-openshift-default-56654dcf7-g96h8\" (UID: \"d8a2a709-efb7-4820-831e-178d5bfb30c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:54:26.824200 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:26.824115 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a2a709-efb7-4820-831e-178d5bfb30c8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-g96h8\" (UID: \"d8a2a709-efb7-4820-831e-178d5bfb30c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:54:26.824200 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:26.824159 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a2a709-efb7-4820-831e-178d5bfb30c8-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-g96h8\" (UID: \"d8a2a709-efb7-4820-831e-178d5bfb30c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:54:26.824331 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:26.824213 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a2a709-efb7-4820-831e-178d5bfb30c8-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-g96h8\" (UID: \"d8a2a709-efb7-4820-831e-178d5bfb30c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:54:26.824331 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:26.824239 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a2a709-efb7-4820-831e-178d5bfb30c8-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-g96h8\" (UID: \"d8a2a709-efb7-4820-831e-178d5bfb30c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:54:26.824331 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:26.824299 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a2a709-efb7-4820-831e-178d5bfb30c8-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-g96h8\" (UID: \"d8a2a709-efb7-4820-831e-178d5bfb30c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:54:26.824472 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:26.824341 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a2a709-efb7-4820-831e-178d5bfb30c8-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-g96h8\" (UID: \"d8a2a709-efb7-4820-831e-178d5bfb30c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:54:26.824602 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:26.824580 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a2a709-efb7-4820-831e-178d5bfb30c8-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-g96h8\" (UID: \"d8a2a709-efb7-4820-831e-178d5bfb30c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:54:26.824696 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:26.824676 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a2a709-efb7-4820-831e-178d5bfb30c8-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-g96h8\" (UID: \"d8a2a709-efb7-4820-831e-178d5bfb30c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:54:26.825040 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:26.825022 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a2a709-efb7-4820-831e-178d5bfb30c8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-g96h8\" (UID: \"d8a2a709-efb7-4820-831e-178d5bfb30c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:54:26.826443 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:26.826422 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a2a709-efb7-4820-831e-178d5bfb30c8-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-g96h8\" (UID: \"d8a2a709-efb7-4820-831e-178d5bfb30c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:54:26.826722 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:26.826701 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a2a709-efb7-4820-831e-178d5bfb30c8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-g96h8\" (UID: \"d8a2a709-efb7-4820-831e-178d5bfb30c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:54:26.832345 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:26.832319 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4s9h\" (UniqueName: \"kubernetes.io/projected/d8a2a709-efb7-4820-831e-178d5bfb30c8-kube-api-access-h4s9h\") pod \"maas-default-gateway-openshift-default-56654dcf7-g96h8\" (UID: \"d8a2a709-efb7-4820-831e-178d5bfb30c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:54:26.832424 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:26.832375 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8a2a709-efb7-4820-831e-178d5bfb30c8-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-g96h8\" (UID: \"d8a2a709-efb7-4820-831e-178d5bfb30c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:54:26.893646 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:26.893619 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:54:27.062360 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:27.062332 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8"] Jul 08 18:54:27.064365 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:54:27.064338 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a2a709_efb7_4820_831e_178d5bfb30c8.slice/crio-4704fc102f46182e31f88e86a64e7a4cccdca8f45abb0db2193ea9e14152d49c WatchSource:0}: Error finding container 4704fc102f46182e31f88e86a64e7a4cccdca8f45abb0db2193ea9e14152d49c: Status 404 returned error can't find the container with id 4704fc102f46182e31f88e86a64e7a4cccdca8f45abb0db2193ea9e14152d49c Jul 08 18:54:27.246897 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:27.246860 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" event={"ID":"d8a2a709-efb7-4820-831e-178d5bfb30c8","Type":"ContainerStarted","Data":"4704fc102f46182e31f88e86a64e7a4cccdca8f45abb0db2193ea9e14152d49c"} Jul 08 18:54:29.961800 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:29.961763 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 18:54:29.962055 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:29.961846 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 18:54:29.962055 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:29.961877 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 18:54:30.276014 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:30.271827 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" event={"ID":"d8a2a709-efb7-4820-831e-178d5bfb30c8","Type":"ContainerStarted","Data":"1eb8104d1e7932af8468f4cacf625da3a84d7dd021838d64c0225486b5103530"} Jul 08 18:54:30.296649 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:30.296597 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" podStartSLOduration=1.401499424 podStartE2EDuration="4.296582468s" totalImagesPullingTime="2.895083044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:54:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:54:30.293533825 +0000 UTC m=+531.140808490" watchObservedRunningTime="2026-07-08 18:54:30.296582468 +0000 UTC m=+531.143857133" Jul 08 18:54:30.894409 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:30.894376 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:54:31.899211 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:31.899182 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:54:32.283904 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:32.283825 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:54:32.284669 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:32.284652 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:54:42.196154 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:42.196120 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hkqxg"] Jul 08 18:54:42.201046 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:42.201028 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-hkqxg" Jul 08 18:54:42.203575 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:42.203549 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 08 18:54:42.210635 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:42.210611 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hkqxg"] Jul 08 18:54:42.254010 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:42.253980 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/398eeb75-eea6-4560-bc0c-ccf613c21696-config-file\") pod \"limitador-limitador-64c94fd64f-hkqxg\" (UID: \"398eeb75-eea6-4560-bc0c-ccf613c21696\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hkqxg" Jul 08 18:54:42.254163 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:42.254060 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r42s5\" (UniqueName: \"kubernetes.io/projected/398eeb75-eea6-4560-bc0c-ccf613c21696-kube-api-access-r42s5\") pod \"limitador-limitador-64c94fd64f-hkqxg\" (UID: \"398eeb75-eea6-4560-bc0c-ccf613c21696\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hkqxg" Jul 08 18:54:42.300494 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:42.300460 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hkqxg"] Jul 08 18:54:42.354856 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:42.354819 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r42s5\" (UniqueName: \"kubernetes.io/projected/398eeb75-eea6-4560-bc0c-ccf613c21696-kube-api-access-r42s5\") pod \"limitador-limitador-64c94fd64f-hkqxg\" (UID: \"398eeb75-eea6-4560-bc0c-ccf613c21696\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hkqxg" Jul 08 18:54:42.355032 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:42.354874 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/398eeb75-eea6-4560-bc0c-ccf613c21696-config-file\") pod \"limitador-limitador-64c94fd64f-hkqxg\" (UID: \"398eeb75-eea6-4560-bc0c-ccf613c21696\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hkqxg" Jul 08 18:54:42.355481 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:42.355456 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/398eeb75-eea6-4560-bc0c-ccf613c21696-config-file\") pod \"limitador-limitador-64c94fd64f-hkqxg\" (UID: \"398eeb75-eea6-4560-bc0c-ccf613c21696\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hkqxg" Jul 08 18:54:42.364071 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:42.364050 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r42s5\" (UniqueName: \"kubernetes.io/projected/398eeb75-eea6-4560-bc0c-ccf613c21696-kube-api-access-r42s5\") pod \"limitador-limitador-64c94fd64f-hkqxg\" (UID: \"398eeb75-eea6-4560-bc0c-ccf613c21696\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hkqxg" Jul 08 18:54:42.513123 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:42.513027 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-hkqxg" Jul 08 18:54:42.639598 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:42.639567 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 18:54:42.644183 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:42.644162 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-rg45r" Jul 08 18:54:42.657979 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:42.657800 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9fgc\" (UniqueName: \"kubernetes.io/projected/9bf4e223-1f2a-4b18-82c4-47a92a81eb70-kube-api-access-h9fgc\") pod \"limitador-limitador-54d6d988d5-rg45r\" (UID: \"9bf4e223-1f2a-4b18-82c4-47a92a81eb70\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-rg45r" Jul 08 18:54:42.657979 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:42.657855 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9bf4e223-1f2a-4b18-82c4-47a92a81eb70-config-file\") pod \"limitador-limitador-54d6d988d5-rg45r\" (UID: \"9bf4e223-1f2a-4b18-82c4-47a92a81eb70\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-rg45r" Jul 08 18:54:42.658360 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:42.658209 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 18:54:42.662236 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:42.662215 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hkqxg"] Jul 08 18:54:42.665400 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:54:42.665376 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod398eeb75_eea6_4560_bc0c_ccf613c21696.slice/crio-4dc6fb9b6ccc2ee20fa9cf0f0fd10fa1d6a9a07d4be344b2d832ad88c860f83b WatchSource:0}: Error finding container 4dc6fb9b6ccc2ee20fa9cf0f0fd10fa1d6a9a07d4be344b2d832ad88c860f83b: Status 404 returned error can't find the container with id 4dc6fb9b6ccc2ee20fa9cf0f0fd10fa1d6a9a07d4be344b2d832ad88c860f83b Jul 08 18:54:42.694947 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:42.694917 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 18:54:42.759085 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:42.759054 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9fgc\" (UniqueName: \"kubernetes.io/projected/9bf4e223-1f2a-4b18-82c4-47a92a81eb70-kube-api-access-h9fgc\") pod \"limitador-limitador-54d6d988d5-rg45r\" (UID: \"9bf4e223-1f2a-4b18-82c4-47a92a81eb70\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-rg45r" Jul 08 18:54:42.759288 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:42.759093 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9bf4e223-1f2a-4b18-82c4-47a92a81eb70-config-file\") pod \"limitador-limitador-54d6d988d5-rg45r\" (UID: \"9bf4e223-1f2a-4b18-82c4-47a92a81eb70\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-rg45r" Jul 08 18:54:42.759846 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:42.759827 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9bf4e223-1f2a-4b18-82c4-47a92a81eb70-config-file\") pod \"limitador-limitador-54d6d988d5-rg45r\" (UID: \"9bf4e223-1f2a-4b18-82c4-47a92a81eb70\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-rg45r" Jul 08 18:54:42.768193 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:42.768120 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9fgc\" (UniqueName: \"kubernetes.io/projected/9bf4e223-1f2a-4b18-82c4-47a92a81eb70-kube-api-access-h9fgc\") pod \"limitador-limitador-54d6d988d5-rg45r\" (UID: \"9bf4e223-1f2a-4b18-82c4-47a92a81eb70\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-rg45r" Jul 08 18:54:42.960769 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:42.960728 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-rg45r" Jul 08 18:54:43.100285 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:43.100239 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 18:54:43.101772 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:54:43.101750 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bf4e223_1f2a_4b18_82c4_47a92a81eb70.slice/crio-1361c8a48a274034857a5084c26c6cf34b3963a031ab0a47280b3d7409a49bd4 WatchSource:0}: Error finding container 1361c8a48a274034857a5084c26c6cf34b3963a031ab0a47280b3d7409a49bd4: Status 404 returned error can't find the container with id 1361c8a48a274034857a5084c26c6cf34b3963a031ab0a47280b3d7409a49bd4 Jul 08 18:54:43.346101 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:43.346013 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-hkqxg" event={"ID":"398eeb75-eea6-4560-bc0c-ccf613c21696","Type":"ContainerStarted","Data":"4dc6fb9b6ccc2ee20fa9cf0f0fd10fa1d6a9a07d4be344b2d832ad88c860f83b"} Jul 08 18:54:43.347290 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:43.347266 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-rg45r" event={"ID":"9bf4e223-1f2a-4b18-82c4-47a92a81eb70","Type":"ContainerStarted","Data":"1361c8a48a274034857a5084c26c6cf34b3963a031ab0a47280b3d7409a49bd4"} Jul 08 18:54:46.370552 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:46.370518 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-rg45r" event={"ID":"9bf4e223-1f2a-4b18-82c4-47a92a81eb70","Type":"ContainerStarted","Data":"d8c11cb181e4e640a32082d0857378448bec84781d3d44a9a5d0a038cb151183"} Jul 08 18:54:46.371032 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:46.370689 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-rg45r" Jul 08 18:54:46.372022 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:46.371999 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-hkqxg" event={"ID":"398eeb75-eea6-4560-bc0c-ccf613c21696","Type":"ContainerStarted","Data":"b8589b26666d93ad98f2e7df98b72a4a369d63834701fdf921917f0b461994f6"} Jul 08 18:54:46.372149 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:46.372129 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-hkqxg" Jul 08 18:54:46.396939 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:46.396846 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-rg45r" podStartSLOduration=1.34994502 podStartE2EDuration="4.396830585s" totalImagesPullingTime="3.046885565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:54:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:54:46.394922785 +0000 UTC m=+547.242197452" watchObservedRunningTime="2026-07-08 18:54:46.396830585 +0000 UTC m=+547.244105250" Jul 08 18:54:46.417971 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:46.417920 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-hkqxg" podStartSLOduration=0.927521372 podStartE2EDuration="4.417905545s" totalImagesPullingTime="3.490384173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:54:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:54:46.415848578 +0000 UTC m=+547.263123250" watchObservedRunningTime="2026-07-08 18:54:46.417905545 +0000 UTC m=+547.265180212" Jul 08 18:54:57.379057 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:57.379029 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-rg45r" Jul 08 18:54:57.379529 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:57.379095 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-hkqxg" Jul 08 18:54:57.449219 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:57.449179 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hkqxg"] Jul 08 18:54:57.449420 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:57.449389 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-hkqxg" podUID="398eeb75-eea6-4560-bc0c-ccf613c21696" containerName="limitador" containerID="cri-o://b8589b26666d93ad98f2e7df98b72a4a369d63834701fdf921917f0b461994f6" gracePeriod=30 Jul 08 18:54:57.997624 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:57.997595 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-hkqxg" Jul 08 18:54:58.100673 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:58.100573 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r42s5\" (UniqueName: \"kubernetes.io/projected/398eeb75-eea6-4560-bc0c-ccf613c21696-kube-api-access-r42s5\") pod \"398eeb75-eea6-4560-bc0c-ccf613c21696\" (UID: \"398eeb75-eea6-4560-bc0c-ccf613c21696\") " Jul 08 18:54:58.100853 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:58.100684 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/398eeb75-eea6-4560-bc0c-ccf613c21696-config-file\") pod \"398eeb75-eea6-4560-bc0c-ccf613c21696\" (UID: \"398eeb75-eea6-4560-bc0c-ccf613c21696\") " Jul 08 18:54:58.101066 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:58.101038 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/398eeb75-eea6-4560-bc0c-ccf613c21696-config-file" (OuterVolumeSpecName: "config-file") pod "398eeb75-eea6-4560-bc0c-ccf613c21696" (UID: "398eeb75-eea6-4560-bc0c-ccf613c21696"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:54:58.102805 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:58.102784 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/398eeb75-eea6-4560-bc0c-ccf613c21696-kube-api-access-r42s5" (OuterVolumeSpecName: "kube-api-access-r42s5") pod "398eeb75-eea6-4560-bc0c-ccf613c21696" (UID: "398eeb75-eea6-4560-bc0c-ccf613c21696"). InnerVolumeSpecName "kube-api-access-r42s5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:54:58.202074 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:58.202037 2570 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/398eeb75-eea6-4560-bc0c-ccf613c21696-config-file\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:54:58.202074 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:58.202072 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r42s5\" (UniqueName: \"kubernetes.io/projected/398eeb75-eea6-4560-bc0c-ccf613c21696-kube-api-access-r42s5\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:54:58.437042 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:58.437009 2570 generic.go:356] "Generic (PLEG): container finished" podID="398eeb75-eea6-4560-bc0c-ccf613c21696" containerID="b8589b26666d93ad98f2e7df98b72a4a369d63834701fdf921917f0b461994f6" exitCode=0 Jul 08 18:54:58.437517 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:58.437070 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-hkqxg" Jul 08 18:54:58.437517 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:58.437080 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-hkqxg" event={"ID":"398eeb75-eea6-4560-bc0c-ccf613c21696","Type":"ContainerDied","Data":"b8589b26666d93ad98f2e7df98b72a4a369d63834701fdf921917f0b461994f6"} Jul 08 18:54:58.437517 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:58.437153 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-hkqxg" event={"ID":"398eeb75-eea6-4560-bc0c-ccf613c21696","Type":"ContainerDied","Data":"4dc6fb9b6ccc2ee20fa9cf0f0fd10fa1d6a9a07d4be344b2d832ad88c860f83b"} Jul 08 18:54:58.437517 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:58.437176 2570 scope.go:117] "RemoveContainer" containerID="b8589b26666d93ad98f2e7df98b72a4a369d63834701fdf921917f0b461994f6" Jul 08 18:54:58.446579 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:58.446560 2570 scope.go:117] "RemoveContainer" containerID="b8589b26666d93ad98f2e7df98b72a4a369d63834701fdf921917f0b461994f6" Jul 08 18:54:58.446801 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:54:58.446780 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8589b26666d93ad98f2e7df98b72a4a369d63834701fdf921917f0b461994f6\": container with ID starting with b8589b26666d93ad98f2e7df98b72a4a369d63834701fdf921917f0b461994f6 not found: ID does not exist" containerID="b8589b26666d93ad98f2e7df98b72a4a369d63834701fdf921917f0b461994f6" Jul 08 18:54:58.446864 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:58.446810 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8589b26666d93ad98f2e7df98b72a4a369d63834701fdf921917f0b461994f6"} err="failed to get container status \"b8589b26666d93ad98f2e7df98b72a4a369d63834701fdf921917f0b461994f6\": rpc error: code = NotFound desc = could not find container \"b8589b26666d93ad98f2e7df98b72a4a369d63834701fdf921917f0b461994f6\": container with ID starting with b8589b26666d93ad98f2e7df98b72a4a369d63834701fdf921917f0b461994f6 not found: ID does not exist" Jul 08 18:54:58.466456 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:58.466421 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hkqxg"] Jul 08 18:54:58.477232 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:58.477203 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hkqxg"] Jul 08 18:54:59.690805 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:54:59.690770 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="398eeb75-eea6-4560-bc0c-ccf613c21696" path="/var/lib/kubelet/pods/398eeb75-eea6-4560-bc0c-ccf613c21696/volumes" Jul 08 18:55:03.294725 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:03.294694 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-psr8x"] Jul 08 18:55:03.295112 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:03.295070 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="398eeb75-eea6-4560-bc0c-ccf613c21696" containerName="limitador" Jul 08 18:55:03.295112 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:03.295081 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="398eeb75-eea6-4560-bc0c-ccf613c21696" containerName="limitador" Jul 08 18:55:03.295202 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:03.295142 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="398eeb75-eea6-4560-bc0c-ccf613c21696" containerName="limitador" Jul 08 18:55:03.298280 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:03.298245 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-psr8x" Jul 08 18:55:03.301171 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:03.301148 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 08 18:55:03.301363 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:03.301341 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-rnqp9\"" Jul 08 18:55:03.302317 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:03.302299 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 08 18:55:03.302428 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:03.302359 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 08 18:55:03.307591 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:03.307568 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-psr8x"] Jul 08 18:55:03.351891 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:03.351856 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jpx5\" (UniqueName: \"kubernetes.io/projected/895a8435-57cf-40f0-8aac-d98867ba0a47-kube-api-access-4jpx5\") pod \"postgres-69fb4594cf-psr8x\" (UID: \"895a8435-57cf-40f0-8aac-d98867ba0a47\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-psr8x" Jul 08 18:55:03.351891 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:03.351900 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/895a8435-57cf-40f0-8aac-d98867ba0a47-data\") pod \"postgres-69fb4594cf-psr8x\" (UID: \"895a8435-57cf-40f0-8aac-d98867ba0a47\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-psr8x" Jul 08 18:55:03.453210 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:03.453168 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jpx5\" (UniqueName: \"kubernetes.io/projected/895a8435-57cf-40f0-8aac-d98867ba0a47-kube-api-access-4jpx5\") pod \"postgres-69fb4594cf-psr8x\" (UID: \"895a8435-57cf-40f0-8aac-d98867ba0a47\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-psr8x" Jul 08 18:55:03.453421 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:03.453221 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/895a8435-57cf-40f0-8aac-d98867ba0a47-data\") pod \"postgres-69fb4594cf-psr8x\" (UID: \"895a8435-57cf-40f0-8aac-d98867ba0a47\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-psr8x" Jul 08 18:55:03.453652 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:03.453632 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/895a8435-57cf-40f0-8aac-d98867ba0a47-data\") pod \"postgres-69fb4594cf-psr8x\" (UID: \"895a8435-57cf-40f0-8aac-d98867ba0a47\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-psr8x" Jul 08 18:55:03.463142 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:03.463117 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jpx5\" (UniqueName: \"kubernetes.io/projected/895a8435-57cf-40f0-8aac-d98867ba0a47-kube-api-access-4jpx5\") pod \"postgres-69fb4594cf-psr8x\" (UID: \"895a8435-57cf-40f0-8aac-d98867ba0a47\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-psr8x" Jul 08 18:55:03.609816 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:03.609786 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-psr8x" Jul 08 18:55:03.749833 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:03.749801 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-psr8x"] Jul 08 18:55:03.750703 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:55:03.750672 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod895a8435_57cf_40f0_8aac_d98867ba0a47.slice/crio-0e5338146948bc90b7571028e604d8ea8dcdf37e805c1b069005a5c12140feb7 WatchSource:0}: Error finding container 0e5338146948bc90b7571028e604d8ea8dcdf37e805c1b069005a5c12140feb7: Status 404 returned error can't find the container with id 0e5338146948bc90b7571028e604d8ea8dcdf37e805c1b069005a5c12140feb7 Jul 08 18:55:04.470661 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:04.470626 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-psr8x" event={"ID":"895a8435-57cf-40f0-8aac-d98867ba0a47","Type":"ContainerStarted","Data":"0e5338146948bc90b7571028e604d8ea8dcdf37e805c1b069005a5c12140feb7"} Jul 08 18:55:09.509201 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:09.509160 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-psr8x" event={"ID":"895a8435-57cf-40f0-8aac-d98867ba0a47","Type":"ContainerStarted","Data":"a203390f22450f2bf31e563937aa6dde8f375af821dbb18011ef222ba1847eb7"} Jul 08 18:55:09.509665 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:09.509280 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-psr8x" Jul 08 18:55:09.529244 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:09.529197 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-psr8x" podStartSLOduration=1.7165600589999999 podStartE2EDuration="6.529182097s" totalImagesPullingTime="4.812622038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:55:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:55:09.526722526 +0000 UTC m=+570.373997193" watchObservedRunningTime="2026-07-08 18:55:09.529182097 +0000 UTC m=+570.376456763" Jul 08 18:55:15.544050 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:15.544019 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-psr8x" Jul 08 18:55:23.239686 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:23.239648 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-b88dcff6f-h25qj"] Jul 08 18:55:23.248459 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:23.248438 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b88dcff6f-h25qj" Jul 08 18:55:23.251571 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:23.251546 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 08 18:55:23.252459 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:23.252442 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 08 18:55:23.252532 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:23.252451 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-v7xtb\"" Jul 08 18:55:23.279733 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:23.279704 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b88dcff6f-h25qj"] Jul 08 18:55:23.439085 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:23.439050 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4ppd\" (UniqueName: \"kubernetes.io/projected/fd6babd7-6dee-45e9-8a67-ced74f33728a-kube-api-access-s4ppd\") pod \"maas-controller-b88dcff6f-h25qj\" (UID: \"fd6babd7-6dee-45e9-8a67-ced74f33728a\") " pod="opendatahub/maas-controller-b88dcff6f-h25qj" Jul 08 18:55:23.439235 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:23.439096 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd6babd7-6dee-45e9-8a67-ced74f33728a-webhook-cert\") pod \"maas-controller-b88dcff6f-h25qj\" (UID: \"fd6babd7-6dee-45e9-8a67-ced74f33728a\") " pod="opendatahub/maas-controller-b88dcff6f-h25qj" Jul 08 18:55:23.504300 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:23.504218 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6db59b6dbc-j8s7w"] Jul 08 18:55:23.508125 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:23.508110 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6db59b6dbc-j8s7w" Jul 08 18:55:23.519927 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:23.519906 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6db59b6dbc-j8s7w"] Jul 08 18:55:23.539932 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:23.539909 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4ppd\" (UniqueName: \"kubernetes.io/projected/fd6babd7-6dee-45e9-8a67-ced74f33728a-kube-api-access-s4ppd\") pod \"maas-controller-b88dcff6f-h25qj\" (UID: \"fd6babd7-6dee-45e9-8a67-ced74f33728a\") " pod="opendatahub/maas-controller-b88dcff6f-h25qj" Jul 08 18:55:23.540059 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:23.539946 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd6babd7-6dee-45e9-8a67-ced74f33728a-webhook-cert\") pod \"maas-controller-b88dcff6f-h25qj\" (UID: \"fd6babd7-6dee-45e9-8a67-ced74f33728a\") " pod="opendatahub/maas-controller-b88dcff6f-h25qj" Jul 08 18:55:23.540144 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:55:23.540128 2570 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 08 18:55:23.540197 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:55:23.540187 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd6babd7-6dee-45e9-8a67-ced74f33728a-webhook-cert podName:fd6babd7-6dee-45e9-8a67-ced74f33728a nodeName:}" failed. No retries permitted until 2026-07-08 18:55:24.040171567 +0000 UTC m=+584.887446212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/fd6babd7-6dee-45e9-8a67-ced74f33728a-webhook-cert") pod "maas-controller-b88dcff6f-h25qj" (UID: "fd6babd7-6dee-45e9-8a67-ced74f33728a") : secret "maas-controller-webhook-cert" not found Jul 08 18:55:23.558182 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:23.558157 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4ppd\" (UniqueName: \"kubernetes.io/projected/fd6babd7-6dee-45e9-8a67-ced74f33728a-kube-api-access-s4ppd\") pod \"maas-controller-b88dcff6f-h25qj\" (UID: \"fd6babd7-6dee-45e9-8a67-ced74f33728a\") " pod="opendatahub/maas-controller-b88dcff6f-h25qj" Jul 08 18:55:23.641398 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:23.641369 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rm9l\" (UniqueName: \"kubernetes.io/projected/4ce9811e-5b59-40f6-9335-e622fe293cef-kube-api-access-4rm9l\") pod \"maas-controller-6db59b6dbc-j8s7w\" (UID: \"4ce9811e-5b59-40f6-9335-e622fe293cef\") " pod="opendatahub/maas-controller-6db59b6dbc-j8s7w" Jul 08 18:55:23.641524 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:23.641411 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ce9811e-5b59-40f6-9335-e622fe293cef-webhook-cert\") pod \"maas-controller-6db59b6dbc-j8s7w\" (UID: \"4ce9811e-5b59-40f6-9335-e622fe293cef\") " pod="opendatahub/maas-controller-6db59b6dbc-j8s7w" Jul 08 18:55:23.742541 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:23.742509 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rm9l\" (UniqueName: \"kubernetes.io/projected/4ce9811e-5b59-40f6-9335-e622fe293cef-kube-api-access-4rm9l\") pod \"maas-controller-6db59b6dbc-j8s7w\" (UID: \"4ce9811e-5b59-40f6-9335-e622fe293cef\") " pod="opendatahub/maas-controller-6db59b6dbc-j8s7w" Jul 08 18:55:23.742717 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:23.742566 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ce9811e-5b59-40f6-9335-e622fe293cef-webhook-cert\") pod \"maas-controller-6db59b6dbc-j8s7w\" (UID: \"4ce9811e-5b59-40f6-9335-e622fe293cef\") " pod="opendatahub/maas-controller-6db59b6dbc-j8s7w" Jul 08 18:55:23.742783 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:55:23.742750 2570 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 08 18:55:23.742835 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:55:23.742821 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ce9811e-5b59-40f6-9335-e622fe293cef-webhook-cert podName:4ce9811e-5b59-40f6-9335-e622fe293cef nodeName:}" failed. No retries permitted until 2026-07-08 18:55:24.24280172 +0000 UTC m=+585.090076370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/4ce9811e-5b59-40f6-9335-e622fe293cef-webhook-cert") pod "maas-controller-6db59b6dbc-j8s7w" (UID: "4ce9811e-5b59-40f6-9335-e622fe293cef") : secret "maas-controller-webhook-cert" not found Jul 08 18:55:23.754393 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:23.754328 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rm9l\" (UniqueName: \"kubernetes.io/projected/4ce9811e-5b59-40f6-9335-e622fe293cef-kube-api-access-4rm9l\") pod \"maas-controller-6db59b6dbc-j8s7w\" (UID: \"4ce9811e-5b59-40f6-9335-e622fe293cef\") " pod="opendatahub/maas-controller-6db59b6dbc-j8s7w" Jul 08 18:55:24.045105 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:24.045027 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd6babd7-6dee-45e9-8a67-ced74f33728a-webhook-cert\") pod \"maas-controller-b88dcff6f-h25qj\" (UID: \"fd6babd7-6dee-45e9-8a67-ced74f33728a\") " pod="opendatahub/maas-controller-b88dcff6f-h25qj" Jul 08 18:55:24.045272 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:55:24.045195 2570 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 08 18:55:24.045330 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:55:24.045289 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd6babd7-6dee-45e9-8a67-ced74f33728a-webhook-cert podName:fd6babd7-6dee-45e9-8a67-ced74f33728a nodeName:}" failed. No retries permitted until 2026-07-08 18:55:25.045269685 +0000 UTC m=+585.892544335 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/fd6babd7-6dee-45e9-8a67-ced74f33728a-webhook-cert") pod "maas-controller-b88dcff6f-h25qj" (UID: "fd6babd7-6dee-45e9-8a67-ced74f33728a") : secret "maas-controller-webhook-cert" not found Jul 08 18:55:24.247289 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:24.247237 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ce9811e-5b59-40f6-9335-e622fe293cef-webhook-cert\") pod \"maas-controller-6db59b6dbc-j8s7w\" (UID: \"4ce9811e-5b59-40f6-9335-e622fe293cef\") " pod="opendatahub/maas-controller-6db59b6dbc-j8s7w" Jul 08 18:55:24.247646 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:55:24.247392 2570 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 08 18:55:24.247646 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:55:24.247456 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ce9811e-5b59-40f6-9335-e622fe293cef-webhook-cert podName:4ce9811e-5b59-40f6-9335-e622fe293cef nodeName:}" failed. No retries permitted until 2026-07-08 18:55:25.247440815 +0000 UTC m=+586.094715459 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/4ce9811e-5b59-40f6-9335-e622fe293cef-webhook-cert") pod "maas-controller-6db59b6dbc-j8s7w" (UID: "4ce9811e-5b59-40f6-9335-e622fe293cef") : secret "maas-controller-webhook-cert" not found Jul 08 18:55:25.053796 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:25.053767 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd6babd7-6dee-45e9-8a67-ced74f33728a-webhook-cert\") pod \"maas-controller-b88dcff6f-h25qj\" (UID: \"fd6babd7-6dee-45e9-8a67-ced74f33728a\") " pod="opendatahub/maas-controller-b88dcff6f-h25qj" Jul 08 18:55:25.056027 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:25.056008 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd6babd7-6dee-45e9-8a67-ced74f33728a-webhook-cert\") pod \"maas-controller-b88dcff6f-h25qj\" (UID: \"fd6babd7-6dee-45e9-8a67-ced74f33728a\") " pod="opendatahub/maas-controller-b88dcff6f-h25qj" Jul 08 18:55:25.058942 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:25.058923 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b88dcff6f-h25qj" Jul 08 18:55:25.197607 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:25.197578 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b88dcff6f-h25qj"] Jul 08 18:55:25.199616 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:55:25.199587 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd6babd7_6dee_45e9_8a67_ced74f33728a.slice/crio-d7a7f8dbadfea803de3a6f45bc910599cf6f0a008d4f1b7cba7bb1aee2c5bc19 WatchSource:0}: Error finding container d7a7f8dbadfea803de3a6f45bc910599cf6f0a008d4f1b7cba7bb1aee2c5bc19: Status 404 returned error can't find the container with id d7a7f8dbadfea803de3a6f45bc910599cf6f0a008d4f1b7cba7bb1aee2c5bc19 Jul 08 18:55:25.255759 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:25.255734 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ce9811e-5b59-40f6-9335-e622fe293cef-webhook-cert\") pod \"maas-controller-6db59b6dbc-j8s7w\" (UID: \"4ce9811e-5b59-40f6-9335-e622fe293cef\") " pod="opendatahub/maas-controller-6db59b6dbc-j8s7w" Jul 08 18:55:25.257907 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:25.257889 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ce9811e-5b59-40f6-9335-e622fe293cef-webhook-cert\") pod \"maas-controller-6db59b6dbc-j8s7w\" (UID: \"4ce9811e-5b59-40f6-9335-e622fe293cef\") " pod="opendatahub/maas-controller-6db59b6dbc-j8s7w" Jul 08 18:55:25.318305 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:25.318234 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6db59b6dbc-j8s7w" Jul 08 18:55:25.459558 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:25.459535 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6db59b6dbc-j8s7w"] Jul 08 18:55:25.461352 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:55:25.461326 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ce9811e_5b59_40f6_9335_e622fe293cef.slice/crio-73e4cbace267431945bff4247539ffcaef4400689f19ae719ccdda5a86543608 WatchSource:0}: Error finding container 73e4cbace267431945bff4247539ffcaef4400689f19ae719ccdda5a86543608: Status 404 returned error can't find the container with id 73e4cbace267431945bff4247539ffcaef4400689f19ae719ccdda5a86543608 Jul 08 18:55:25.595030 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:25.594955 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b88dcff6f-h25qj" event={"ID":"fd6babd7-6dee-45e9-8a67-ced74f33728a","Type":"ContainerStarted","Data":"d7a7f8dbadfea803de3a6f45bc910599cf6f0a008d4f1b7cba7bb1aee2c5bc19"} Jul 08 18:55:25.596068 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:25.596049 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6db59b6dbc-j8s7w" event={"ID":"4ce9811e-5b59-40f6-9335-e622fe293cef","Type":"ContainerStarted","Data":"73e4cbace267431945bff4247539ffcaef4400689f19ae719ccdda5a86543608"} Jul 08 18:55:27.612030 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:27.611994 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b88dcff6f-h25qj" event={"ID":"fd6babd7-6dee-45e9-8a67-ced74f33728a","Type":"ContainerStarted","Data":"bc184e88051507455ed520f3a58e38f1080f6312312fd3023956120d4a2e0273"} Jul 08 18:55:27.612495 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:27.612057 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-b88dcff6f-h25qj" Jul 08 18:55:27.613684 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:27.613660 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6db59b6dbc-j8s7w" event={"ID":"4ce9811e-5b59-40f6-9335-e622fe293cef","Type":"ContainerStarted","Data":"5f516aba3833ee1f5c8699ea0f2661ee8ddd03b140a78fc5376827d9eaff5bd2"} Jul 08 18:55:27.613830 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:27.613731 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6db59b6dbc-j8s7w" Jul 08 18:55:27.640547 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:27.640491 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-b88dcff6f-h25qj" podStartSLOduration=2.413325184 podStartE2EDuration="4.640473861s" totalImagesPullingTime="2.227148677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:55:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:55:27.637417616 +0000 UTC m=+588.484692283" watchObservedRunningTime="2026-07-08 18:55:27.640473861 +0000 UTC m=+588.487748527" Jul 08 18:55:27.666124 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:27.666073 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6db59b6dbc-j8s7w" podStartSLOduration=2.690055886 podStartE2EDuration="4.666056807s" totalImagesPullingTime="1.976000921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:55:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:55:27.662715237 +0000 UTC m=+588.509989927" watchObservedRunningTime="2026-07-08 18:55:27.666056807 +0000 UTC m=+588.513331472" Jul 08 18:55:38.624842 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:38.624808 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-b88dcff6f-h25qj" Jul 08 18:55:38.625434 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:38.624896 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6db59b6dbc-j8s7w" Jul 08 18:55:38.707990 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:38.707956 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b88dcff6f-h25qj"] Jul 08 18:55:38.708199 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:38.708173 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-b88dcff6f-h25qj" podUID="fd6babd7-6dee-45e9-8a67-ced74f33728a" containerName="manager" containerID="cri-o://bc184e88051507455ed520f3a58e38f1080f6312312fd3023956120d4a2e0273" gracePeriod=30 Jul 08 18:55:38.961144 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:38.961120 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b88dcff6f-h25qj" Jul 08 18:55:39.079967 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:39.079939 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd6babd7-6dee-45e9-8a67-ced74f33728a-webhook-cert\") pod \"fd6babd7-6dee-45e9-8a67-ced74f33728a\" (UID: \"fd6babd7-6dee-45e9-8a67-ced74f33728a\") " Jul 08 18:55:39.080166 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:39.079989 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4ppd\" (UniqueName: \"kubernetes.io/projected/fd6babd7-6dee-45e9-8a67-ced74f33728a-kube-api-access-s4ppd\") pod \"fd6babd7-6dee-45e9-8a67-ced74f33728a\" (UID: \"fd6babd7-6dee-45e9-8a67-ced74f33728a\") " Jul 08 18:55:39.082228 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:39.082189 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd6babd7-6dee-45e9-8a67-ced74f33728a-kube-api-access-s4ppd" (OuterVolumeSpecName: "kube-api-access-s4ppd") pod "fd6babd7-6dee-45e9-8a67-ced74f33728a" (UID: "fd6babd7-6dee-45e9-8a67-ced74f33728a"). InnerVolumeSpecName "kube-api-access-s4ppd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:55:39.082228 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:39.082188 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd6babd7-6dee-45e9-8a67-ced74f33728a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "fd6babd7-6dee-45e9-8a67-ced74f33728a" (UID: "fd6babd7-6dee-45e9-8a67-ced74f33728a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:55:39.181005 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:39.180934 2570 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd6babd7-6dee-45e9-8a67-ced74f33728a-webhook-cert\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:55:39.181005 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:39.180957 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4ppd\" (UniqueName: \"kubernetes.io/projected/fd6babd7-6dee-45e9-8a67-ced74f33728a-kube-api-access-s4ppd\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:55:39.680334 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:39.680302 2570 generic.go:356] "Generic (PLEG): container finished" podID="fd6babd7-6dee-45e9-8a67-ced74f33728a" containerID="bc184e88051507455ed520f3a58e38f1080f6312312fd3023956120d4a2e0273" exitCode=0 Jul 08 18:55:39.680727 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:39.680356 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b88dcff6f-h25qj" Jul 08 18:55:39.680727 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:39.680390 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b88dcff6f-h25qj" event={"ID":"fd6babd7-6dee-45e9-8a67-ced74f33728a","Type":"ContainerDied","Data":"bc184e88051507455ed520f3a58e38f1080f6312312fd3023956120d4a2e0273"} Jul 08 18:55:39.680727 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:39.680475 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b88dcff6f-h25qj" event={"ID":"fd6babd7-6dee-45e9-8a67-ced74f33728a","Type":"ContainerDied","Data":"d7a7f8dbadfea803de3a6f45bc910599cf6f0a008d4f1b7cba7bb1aee2c5bc19"} Jul 08 18:55:39.680727 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:39.680504 2570 scope.go:117] "RemoveContainer" containerID="bc184e88051507455ed520f3a58e38f1080f6312312fd3023956120d4a2e0273" Jul 08 18:55:39.690017 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:39.689993 2570 scope.go:117] "RemoveContainer" containerID="bc184e88051507455ed520f3a58e38f1080f6312312fd3023956120d4a2e0273" Jul 08 18:55:39.690308 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:55:39.690290 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc184e88051507455ed520f3a58e38f1080f6312312fd3023956120d4a2e0273\": container with ID starting with bc184e88051507455ed520f3a58e38f1080f6312312fd3023956120d4a2e0273 not found: ID does not exist" containerID="bc184e88051507455ed520f3a58e38f1080f6312312fd3023956120d4a2e0273" Jul 08 18:55:39.690381 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:39.690317 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc184e88051507455ed520f3a58e38f1080f6312312fd3023956120d4a2e0273"} err="failed to get container status \"bc184e88051507455ed520f3a58e38f1080f6312312fd3023956120d4a2e0273\": rpc error: code = NotFound desc = could not find container \"bc184e88051507455ed520f3a58e38f1080f6312312fd3023956120d4a2e0273\": container with ID starting with bc184e88051507455ed520f3a58e38f1080f6312312fd3023956120d4a2e0273 not found: ID does not exist" Jul 08 18:55:39.706315 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:39.706233 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b88dcff6f-h25qj"] Jul 08 18:55:39.719745 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:39.712354 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-b88dcff6f-h25qj"] Jul 08 18:55:41.690841 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:41.690800 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd6babd7-6dee-45e9-8a67-ced74f33728a" path="/var/lib/kubelet/pods/fd6babd7-6dee-45e9-8a67-ced74f33728a/volumes" Jul 08 18:55:57.897350 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:57.897305 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-56ddff855f-xn5qz"] Jul 08 18:55:57.897804 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:57.897787 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd6babd7-6dee-45e9-8a67-ced74f33728a" containerName="manager" Jul 08 18:55:57.897853 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:57.897807 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd6babd7-6dee-45e9-8a67-ced74f33728a" containerName="manager" Jul 08 18:55:57.897889 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:57.897875 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd6babd7-6dee-45e9-8a67-ced74f33728a" containerName="manager" Jul 08 18:55:57.915083 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:57.915060 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-56ddff855f-xn5qz" Jul 08 18:55:57.917418 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:57.917392 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-56ddff855f-xn5qz"] Jul 08 18:55:57.922745 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:57.922715 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 08 18:55:58.043908 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:58.043877 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09864454-4930-42ef-a97c-81425b6ce40b-maas-api-tls\") pod \"maas-api-56ddff855f-xn5qz\" (UID: \"09864454-4930-42ef-a97c-81425b6ce40b\") " pod="odh-ai-gateway-infra/maas-api-56ddff855f-xn5qz" Jul 08 18:55:58.044104 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:58.043960 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09864454-4930-42ef-a97c-81425b6ce40b-openshift-service-ca\") pod \"maas-api-56ddff855f-xn5qz\" (UID: \"09864454-4930-42ef-a97c-81425b6ce40b\") " pod="odh-ai-gateway-infra/maas-api-56ddff855f-xn5qz" Jul 08 18:55:58.044104 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:58.043997 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzw9d\" (UniqueName: \"kubernetes.io/projected/09864454-4930-42ef-a97c-81425b6ce40b-kube-api-access-gzw9d\") pod \"maas-api-56ddff855f-xn5qz\" (UID: \"09864454-4930-42ef-a97c-81425b6ce40b\") " pod="odh-ai-gateway-infra/maas-api-56ddff855f-xn5qz" Jul 08 18:55:58.145058 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:58.145022 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09864454-4930-42ef-a97c-81425b6ce40b-openshift-service-ca\") pod \"maas-api-56ddff855f-xn5qz\" (UID: \"09864454-4930-42ef-a97c-81425b6ce40b\") " pod="odh-ai-gateway-infra/maas-api-56ddff855f-xn5qz" Jul 08 18:55:58.145058 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:58.145061 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzw9d\" (UniqueName: \"kubernetes.io/projected/09864454-4930-42ef-a97c-81425b6ce40b-kube-api-access-gzw9d\") pod \"maas-api-56ddff855f-xn5qz\" (UID: \"09864454-4930-42ef-a97c-81425b6ce40b\") " pod="odh-ai-gateway-infra/maas-api-56ddff855f-xn5qz" Jul 08 18:55:58.145286 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:58.145121 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09864454-4930-42ef-a97c-81425b6ce40b-maas-api-tls\") pod \"maas-api-56ddff855f-xn5qz\" (UID: \"09864454-4930-42ef-a97c-81425b6ce40b\") " pod="odh-ai-gateway-infra/maas-api-56ddff855f-xn5qz" Jul 08 18:55:58.145286 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:55:58.145267 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 08 18:55:58.145375 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:55:58.145339 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09864454-4930-42ef-a97c-81425b6ce40b-maas-api-tls podName:09864454-4930-42ef-a97c-81425b6ce40b nodeName:}" failed. No retries permitted until 2026-07-08 18:55:58.64531956 +0000 UTC m=+619.492594219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/09864454-4930-42ef-a97c-81425b6ce40b-maas-api-tls") pod "maas-api-56ddff855f-xn5qz" (UID: "09864454-4930-42ef-a97c-81425b6ce40b") : secret "maas-api-serving-cert" not found Jul 08 18:55:58.145672 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:58.145654 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09864454-4930-42ef-a97c-81425b6ce40b-openshift-service-ca\") pod \"maas-api-56ddff855f-xn5qz\" (UID: \"09864454-4930-42ef-a97c-81425b6ce40b\") " pod="odh-ai-gateway-infra/maas-api-56ddff855f-xn5qz" Jul 08 18:55:58.158365 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:58.158307 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzw9d\" (UniqueName: \"kubernetes.io/projected/09864454-4930-42ef-a97c-81425b6ce40b-kube-api-access-gzw9d\") pod \"maas-api-56ddff855f-xn5qz\" (UID: \"09864454-4930-42ef-a97c-81425b6ce40b\") " pod="odh-ai-gateway-infra/maas-api-56ddff855f-xn5qz" Jul 08 18:55:58.622774 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:58.622740 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-gj4nt"] Jul 08 18:55:58.625568 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:58.625545 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 18:55:58.625793 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:58.625766 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-gj4nt" Jul 08 18:55:58.629330 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:58.629311 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-jknc6\"" Jul 08 18:55:58.635284 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:58.635244 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 08 18:55:58.649531 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:58.649508 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09864454-4930-42ef-a97c-81425b6ce40b-maas-api-tls\") pod \"maas-api-56ddff855f-xn5qz\" (UID: \"09864454-4930-42ef-a97c-81425b6ce40b\") " pod="odh-ai-gateway-infra/maas-api-56ddff855f-xn5qz" Jul 08 18:55:58.651805 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:58.651776 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09864454-4930-42ef-a97c-81425b6ce40b-maas-api-tls\") pod \"maas-api-56ddff855f-xn5qz\" (UID: \"09864454-4930-42ef-a97c-81425b6ce40b\") " pod="odh-ai-gateway-infra/maas-api-56ddff855f-xn5qz" Jul 08 18:55:58.655586 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:58.655565 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-gj4nt"] Jul 08 18:55:58.750990 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:58.750960 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ms2w\" (UniqueName: \"kubernetes.io/projected/03edc411-3219-4b5a-8b2b-16d7b9b593f4-kube-api-access-5ms2w\") pod \"payload-processing-58f86c7798-gj4nt\" (UID: \"03edc411-3219-4b5a-8b2b-16d7b9b593f4\") " pod="openshift-ingress/payload-processing-58f86c7798-gj4nt" Jul 08 18:55:58.751169 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:58.751086 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03edc411-3219-4b5a-8b2b-16d7b9b593f4-plugins-config-volume\") pod \"payload-processing-58f86c7798-gj4nt\" (UID: \"03edc411-3219-4b5a-8b2b-16d7b9b593f4\") " pod="openshift-ingress/payload-processing-58f86c7798-gj4nt" Jul 08 18:55:58.826361 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:58.826331 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-56ddff855f-xn5qz" Jul 08 18:55:58.851869 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:58.851840 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03edc411-3219-4b5a-8b2b-16d7b9b593f4-plugins-config-volume\") pod \"payload-processing-58f86c7798-gj4nt\" (UID: \"03edc411-3219-4b5a-8b2b-16d7b9b593f4\") " pod="openshift-ingress/payload-processing-58f86c7798-gj4nt" Jul 08 18:55:58.852013 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:58.851921 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ms2w\" (UniqueName: \"kubernetes.io/projected/03edc411-3219-4b5a-8b2b-16d7b9b593f4-kube-api-access-5ms2w\") pod \"payload-processing-58f86c7798-gj4nt\" (UID: \"03edc411-3219-4b5a-8b2b-16d7b9b593f4\") " pod="openshift-ingress/payload-processing-58f86c7798-gj4nt" Jul 08 18:55:58.852673 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:58.852649 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03edc411-3219-4b5a-8b2b-16d7b9b593f4-plugins-config-volume\") pod \"payload-processing-58f86c7798-gj4nt\" (UID: \"03edc411-3219-4b5a-8b2b-16d7b9b593f4\") " pod="openshift-ingress/payload-processing-58f86c7798-gj4nt" Jul 08 18:55:58.861225 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:58.861192 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ms2w\" (UniqueName: \"kubernetes.io/projected/03edc411-3219-4b5a-8b2b-16d7b9b593f4-kube-api-access-5ms2w\") pod \"payload-processing-58f86c7798-gj4nt\" (UID: \"03edc411-3219-4b5a-8b2b-16d7b9b593f4\") " pod="openshift-ingress/payload-processing-58f86c7798-gj4nt" Jul 08 18:55:58.936823 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:58.936800 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-gj4nt" Jul 08 18:55:58.987218 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:58.987172 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-56ddff855f-xn5qz"] Jul 08 18:55:58.987987 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:55:58.987942 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09864454_4930_42ef_a97c_81425b6ce40b.slice/crio-c60a32d3fd9a0662b89cefd9c6c60033e839b5e74aa5ad8a6bc85e6cff485ed1 WatchSource:0}: Error finding container c60a32d3fd9a0662b89cefd9c6c60033e839b5e74aa5ad8a6bc85e6cff485ed1: Status 404 returned error can't find the container with id c60a32d3fd9a0662b89cefd9c6c60033e839b5e74aa5ad8a6bc85e6cff485ed1 Jul 08 18:55:58.989437 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:58.989414 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 18:55:59.099813 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:59.099772 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-gj4nt"] Jul 08 18:55:59.103948 ip-10-0-130-48 kubenswrapper[2570]: W0708 18:55:59.103916 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03edc411_3219_4b5a_8b2b_16d7b9b593f4.slice/crio-125523c8cb5ede5e1241479b9545d522c3a4e515251c2cdc5b303e1a2b8c6ff0 WatchSource:0}: Error finding container 125523c8cb5ede5e1241479b9545d522c3a4e515251c2cdc5b303e1a2b8c6ff0: Status 404 returned error can't find the container with id 125523c8cb5ede5e1241479b9545d522c3a4e515251c2cdc5b303e1a2b8c6ff0 Jul 08 18:55:59.797148 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:59.797113 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-gj4nt" event={"ID":"03edc411-3219-4b5a-8b2b-16d7b9b593f4","Type":"ContainerStarted","Data":"125523c8cb5ede5e1241479b9545d522c3a4e515251c2cdc5b303e1a2b8c6ff0"} Jul 08 18:55:59.798588 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:55:59.798552 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-56ddff855f-xn5qz" event={"ID":"09864454-4930-42ef-a97c-81425b6ce40b","Type":"ContainerStarted","Data":"c60a32d3fd9a0662b89cefd9c6c60033e839b5e74aa5ad8a6bc85e6cff485ed1"} Jul 08 18:56:01.820985 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:56:01.820937 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-56ddff855f-xn5qz" event={"ID":"09864454-4930-42ef-a97c-81425b6ce40b","Type":"ContainerStarted","Data":"05da4d2c760782d0541d3940a851f071c9762f7a07d0cf9656f9c2f0ebee6232"} Jul 08 18:56:01.821459 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:56:01.821129 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-56ddff855f-xn5qz" Jul 08 18:56:01.822652 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:56:01.822624 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-gj4nt" event={"ID":"03edc411-3219-4b5a-8b2b-16d7b9b593f4","Type":"ContainerStarted","Data":"9269769bc4deb94615af01f20ae0d46c31d813b484f612c6fea6aae978b0cc67"} Jul 08 18:56:01.822793 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:56:01.822751 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-gj4nt" Jul 08 18:56:01.846057 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:56:01.845994 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-56ddff855f-xn5qz" podStartSLOduration=2.5546750339999997 podStartE2EDuration="4.845972351s" totalImagesPullingTime="2.291297317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:55:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:56:01.843529735 +0000 UTC m=+622.690804421" watchObservedRunningTime="2026-07-08 18:56:01.845972351 +0000 UTC m=+622.693247019" Jul 08 18:56:01.861761 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:56:01.861714 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-gj4nt" podStartSLOduration=1.690354744 podStartE2EDuration="3.861696771s" totalImagesPullingTime="2.171342027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:55:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:56:01.861537451 +0000 UTC m=+622.708812118" watchObservedRunningTime="2026-07-08 18:56:01.861696771 +0000 UTC m=+622.708971431" Jul 08 18:56:06.852782 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:56:06.852750 2570 generic.go:356] "Generic (PLEG): container finished" podID="d8a2a709-efb7-4820-831e-178d5bfb30c8" containerID="1eb8104d1e7932af8468f4cacf625da3a84d7dd021838d64c0225486b5103530" exitCode=0 Jul 08 18:56:06.853197 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:56:06.852823 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" event={"ID":"d8a2a709-efb7-4820-831e-178d5bfb30c8","Type":"ContainerDied","Data":"1eb8104d1e7932af8468f4cacf625da3a84d7dd021838d64c0225486b5103530"} Jul 08 18:56:06.853197 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:56:06.853162 2570 scope.go:117] "RemoveContainer" containerID="1eb8104d1e7932af8468f4cacf625da3a84d7dd021838d64c0225486b5103530" Jul 08 18:56:06.854152 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:56:06.854125 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 18:56:06.854211 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:56:06.854191 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 18:56:06.854245 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:56:06.854222 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 18:56:07.843562 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:56:07.843530 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-56ddff855f-xn5qz" Jul 08 18:56:07.865584 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:56:07.865544 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" event={"ID":"d8a2a709-efb7-4820-831e-178d5bfb30c8","Type":"ContainerStarted","Data":"aa44ab1991485c0e25d711f2b7178159ac11f126e156d468d0dfb2d21e69e8b6"} Jul 08 18:56:07.894166 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:56:07.894124 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:56:07.894411 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:56:07.894386 2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-g96h8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 08 18:56:07.894475 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:56:07.894449 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" podUID="d8a2a709-efb7-4820-831e-178d5bfb30c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 08 18:56:08.895009 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:56:08.894979 2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-g96h8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 08 18:56:08.895476 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:56:08.895036 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" podUID="d8a2a709-efb7-4820-831e-178d5bfb30c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 08 18:56:09.894346 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:56:09.894313 2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-g96h8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 08 18:56:09.894632 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:56:09.894370 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" podUID="d8a2a709-efb7-4820-831e-178d5bfb30c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 08 18:56:10.894470 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:56:10.894437 2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-g96h8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 08 18:56:10.894850 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:56:10.894492 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" podUID="d8a2a709-efb7-4820-831e-178d5bfb30c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 08 18:56:11.897699 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:56:11.897672 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:56:11.898104 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:56:11.897939 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:56:11.898581 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:56:11.898564 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g96h8" Jul 08 18:56:12.829926 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:56:12.829902 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-gj4nt" Jul 08 18:56:26.752409 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:56:26.752374 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 18:56:32.921237 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:56:32.921199 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 18:56:45.719335 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:56:45.719301 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 18:56:57.898816 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:56:57.898773 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 18:57:10.011976 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:57:10.011940 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 18:57:20.323958 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:57:20.323919 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 18:59:01.234746 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:01.234664 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 18:59:10.698378 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:10.698344 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 18:59:22.795160 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:22.795121 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 18:59:30.802960 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:30.802925 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 18:59:41.512932 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:41.512896 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 18:59:43.731013 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:43.730974 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6db59b6dbc-j8s7w"] Jul 08 18:59:43.731555 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:43.731205 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-6db59b6dbc-j8s7w" podUID="4ce9811e-5b59-40f6-9335-e622fe293cef" containerName="manager" containerID="cri-o://5f516aba3833ee1f5c8699ea0f2661ee8ddd03b140a78fc5376827d9eaff5bd2" gracePeriod=30 Jul 08 18:59:43.987642 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:43.987577 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6db59b6dbc-j8s7w" Jul 08 18:59:44.098896 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:44.098864 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ce9811e-5b59-40f6-9335-e622fe293cef-webhook-cert\") pod \"4ce9811e-5b59-40f6-9335-e622fe293cef\" (UID: \"4ce9811e-5b59-40f6-9335-e622fe293cef\") " Jul 08 18:59:44.099082 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:44.098974 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rm9l\" (UniqueName: \"kubernetes.io/projected/4ce9811e-5b59-40f6-9335-e622fe293cef-kube-api-access-4rm9l\") pod \"4ce9811e-5b59-40f6-9335-e622fe293cef\" (UID: \"4ce9811e-5b59-40f6-9335-e622fe293cef\") " Jul 08 18:59:44.100868 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:44.100842 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ce9811e-5b59-40f6-9335-e622fe293cef-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "4ce9811e-5b59-40f6-9335-e622fe293cef" (UID: "4ce9811e-5b59-40f6-9335-e622fe293cef"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:59:44.100977 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:44.100929 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ce9811e-5b59-40f6-9335-e622fe293cef-kube-api-access-4rm9l" (OuterVolumeSpecName: "kube-api-access-4rm9l") pod "4ce9811e-5b59-40f6-9335-e622fe293cef" (UID: "4ce9811e-5b59-40f6-9335-e622fe293cef"). InnerVolumeSpecName "kube-api-access-4rm9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:59:44.108673 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:44.108645 2570 generic.go:356] "Generic (PLEG): container finished" podID="4ce9811e-5b59-40f6-9335-e622fe293cef" containerID="5f516aba3833ee1f5c8699ea0f2661ee8ddd03b140a78fc5376827d9eaff5bd2" exitCode=0 Jul 08 18:59:44.108802 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:44.108696 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6db59b6dbc-j8s7w" Jul 08 18:59:44.108802 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:44.108735 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6db59b6dbc-j8s7w" event={"ID":"4ce9811e-5b59-40f6-9335-e622fe293cef","Type":"ContainerDied","Data":"5f516aba3833ee1f5c8699ea0f2661ee8ddd03b140a78fc5376827d9eaff5bd2"} Jul 08 18:59:44.108925 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:44.108802 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6db59b6dbc-j8s7w" event={"ID":"4ce9811e-5b59-40f6-9335-e622fe293cef","Type":"ContainerDied","Data":"73e4cbace267431945bff4247539ffcaef4400689f19ae719ccdda5a86543608"} Jul 08 18:59:44.108925 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:44.108828 2570 scope.go:117] "RemoveContainer" containerID="5f516aba3833ee1f5c8699ea0f2661ee8ddd03b140a78fc5376827d9eaff5bd2" Jul 08 18:59:44.120977 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:44.120958 2570 scope.go:117] "RemoveContainer" containerID="5f516aba3833ee1f5c8699ea0f2661ee8ddd03b140a78fc5376827d9eaff5bd2" Jul 08 18:59:44.121278 ip-10-0-130-48 kubenswrapper[2570]: E0708 18:59:44.121238 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f516aba3833ee1f5c8699ea0f2661ee8ddd03b140a78fc5376827d9eaff5bd2\": container with ID starting with 5f516aba3833ee1f5c8699ea0f2661ee8ddd03b140a78fc5376827d9eaff5bd2 not found: ID does not exist" containerID="5f516aba3833ee1f5c8699ea0f2661ee8ddd03b140a78fc5376827d9eaff5bd2" Jul 08 18:59:44.121348 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:44.121288 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f516aba3833ee1f5c8699ea0f2661ee8ddd03b140a78fc5376827d9eaff5bd2"} err="failed to get container status \"5f516aba3833ee1f5c8699ea0f2661ee8ddd03b140a78fc5376827d9eaff5bd2\": rpc error: code = NotFound desc = could not find container \"5f516aba3833ee1f5c8699ea0f2661ee8ddd03b140a78fc5376827d9eaff5bd2\": container with ID starting with 5f516aba3833ee1f5c8699ea0f2661ee8ddd03b140a78fc5376827d9eaff5bd2 not found: ID does not exist" Jul 08 18:59:44.138627 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:44.138597 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6db59b6dbc-j8s7w"] Jul 08 18:59:44.145636 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:44.145613 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6db59b6dbc-j8s7w"] Jul 08 18:59:44.199707 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:44.199676 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rm9l\" (UniqueName: \"kubernetes.io/projected/4ce9811e-5b59-40f6-9335-e622fe293cef-kube-api-access-4rm9l\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:59:44.199707 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:44.199700 2570 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ce9811e-5b59-40f6-9335-e622fe293cef-webhook-cert\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 18:59:45.029195 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:45.029162 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6db59b6dbc-q7l65"] Jul 08 18:59:45.029807 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:45.029787 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ce9811e-5b59-40f6-9335-e622fe293cef" containerName="manager" Jul 08 18:59:45.029901 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:45.029810 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ce9811e-5b59-40f6-9335-e622fe293cef" containerName="manager" Jul 08 18:59:45.029965 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:45.029907 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ce9811e-5b59-40f6-9335-e622fe293cef" containerName="manager" Jul 08 18:59:45.034395 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:45.034374 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6db59b6dbc-q7l65" Jul 08 18:59:45.038565 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:45.038546 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 08 18:59:45.038679 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:45.038635 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 08 18:59:45.040434 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:45.040409 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-v7xtb\"" Jul 08 18:59:45.062342 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:45.062310 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6db59b6dbc-q7l65"] Jul 08 18:59:45.108351 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:45.108322 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92caa313-e898-4633-b6f6-fb6b24aae095-webhook-cert\") pod \"maas-controller-6db59b6dbc-q7l65\" (UID: \"92caa313-e898-4633-b6f6-fb6b24aae095\") " pod="opendatahub/maas-controller-6db59b6dbc-q7l65" Jul 08 18:59:45.108525 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:45.108467 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfgzs\" (UniqueName: \"kubernetes.io/projected/92caa313-e898-4633-b6f6-fb6b24aae095-kube-api-access-cfgzs\") pod \"maas-controller-6db59b6dbc-q7l65\" (UID: \"92caa313-e898-4633-b6f6-fb6b24aae095\") " pod="opendatahub/maas-controller-6db59b6dbc-q7l65" Jul 08 18:59:45.209741 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:45.209689 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfgzs\" (UniqueName: \"kubernetes.io/projected/92caa313-e898-4633-b6f6-fb6b24aae095-kube-api-access-cfgzs\") pod \"maas-controller-6db59b6dbc-q7l65\" (UID: \"92caa313-e898-4633-b6f6-fb6b24aae095\") " pod="opendatahub/maas-controller-6db59b6dbc-q7l65" Jul 08 18:59:45.209928 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:45.209734 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92caa313-e898-4633-b6f6-fb6b24aae095-webhook-cert\") pod \"maas-controller-6db59b6dbc-q7l65\" (UID: \"92caa313-e898-4633-b6f6-fb6b24aae095\") " pod="opendatahub/maas-controller-6db59b6dbc-q7l65" Jul 08 18:59:45.212225 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:45.212198 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92caa313-e898-4633-b6f6-fb6b24aae095-webhook-cert\") pod \"maas-controller-6db59b6dbc-q7l65\" (UID: \"92caa313-e898-4633-b6f6-fb6b24aae095\") " pod="opendatahub/maas-controller-6db59b6dbc-q7l65" Jul 08 18:59:45.223026 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:45.222990 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfgzs\" (UniqueName: \"kubernetes.io/projected/92caa313-e898-4633-b6f6-fb6b24aae095-kube-api-access-cfgzs\") pod \"maas-controller-6db59b6dbc-q7l65\" (UID: \"92caa313-e898-4633-b6f6-fb6b24aae095\") " pod="opendatahub/maas-controller-6db59b6dbc-q7l65" Jul 08 18:59:45.345614 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:45.345514 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6db59b6dbc-q7l65" Jul 08 18:59:45.495689 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:45.491857 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6db59b6dbc-q7l65"] Jul 08 18:59:45.689968 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:45.689933 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ce9811e-5b59-40f6-9335-e622fe293cef" path="/var/lib/kubelet/pods/4ce9811e-5b59-40f6-9335-e622fe293cef/volumes" Jul 08 18:59:46.122105 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:46.122059 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6db59b6dbc-q7l65" event={"ID":"92caa313-e898-4633-b6f6-fb6b24aae095","Type":"ContainerStarted","Data":"fce6e2f247058daf0a85c8a396be9d4e4639e331cd40c366d1880fd18c2d2fb9"} Jul 08 18:59:47.130276 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:47.130221 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6db59b6dbc-q7l65" event={"ID":"92caa313-e898-4633-b6f6-fb6b24aae095","Type":"ContainerStarted","Data":"421fb3335b67517ebdbb8d84d469fa20f9eb67dbee38270a2f8d3f3223ee0432"} Jul 08 18:59:47.130989 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:47.130305 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6db59b6dbc-q7l65" Jul 08 18:59:58.143451 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:58.143414 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6db59b6dbc-q7l65" Jul 08 18:59:58.174771 ip-10-0-130-48 kubenswrapper[2570]: I0708 18:59:58.174714 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6db59b6dbc-q7l65" podStartSLOduration=12.533136993 podStartE2EDuration="13.174696396s" totalImagesPullingTime="641.559403ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:59:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:59:47.162754241 +0000 UTC m=+848.010028920" watchObservedRunningTime="2026-07-08 18:59:58.174696396 +0000 UTC m=+859.021971061" Jul 08 19:00:00.161596 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:00:00.161554 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725620-z86z2"] Jul 08 19:00:00.165402 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:00:00.165384 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725620-z86z2" Jul 08 19:00:00.168563 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:00:00.168543 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m2spm\"" Jul 08 19:00:00.175990 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:00:00.175966 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725620-z86z2" Jul 08 19:00:00.177807 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:00:00.177782 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725620-z86z2"] Jul 08 19:00:00.320628 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:00:00.320602 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725620-z86z2"] Jul 08 19:00:00.322651 ip-10-0-130-48 kubenswrapper[2570]: W0708 19:00:00.322616 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b41b382_a14c_4f73_a3f6_59cf83900ad3.slice/crio-04be29473376b07f6cda4913fe823e29c9de729db42c491635d31e7658d07c9e WatchSource:0}: Error finding container 04be29473376b07f6cda4913fe823e29c9de729db42c491635d31e7658d07c9e: Status 404 returned error can't find the container with id 04be29473376b07f6cda4913fe823e29c9de729db42c491635d31e7658d07c9e Jul 08 19:00:01.210812 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:00:01.210775 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725620-z86z2" event={"ID":"2b41b382-a14c-4f73-a3f6-59cf83900ad3","Type":"ContainerStarted","Data":"04be29473376b07f6cda4913fe823e29c9de729db42c491635d31e7658d07c9e"} Jul 08 19:00:06.247323 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:00:06.247287 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725620-z86z2" event={"ID":"2b41b382-a14c-4f73-a3f6-59cf83900ad3","Type":"ContainerStarted","Data":"83fb6231506f8d27329a8c3263ba431662c60dd205de8bb0cf7485d4d12b3351"} Jul 08 19:00:06.271595 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:00:06.271547 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725620-z86z2" podStartSLOduration=1.170476738 podStartE2EDuration="6.271532222s" totalImagesPullingTime="5.101055484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:00:06.271107856 +0000 UTC m=+867.118382523" watchObservedRunningTime="2026-07-08 19:00:06.271532222 +0000 UTC m=+867.118806888" Jul 08 19:00:10.842043 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:00:10.842005 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:00:23.225424 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:00:23.225385 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:00:58.714371 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:00:58.714329 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:01:00.012053 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:01:00.012018 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725620-z86z2"] Jul 08 19:01:00.012532 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:01:00.012206 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725620-z86z2" podUID="2b41b382-a14c-4f73-a3f6-59cf83900ad3" containerName="cleanup" containerID="cri-o://83fb6231506f8d27329a8c3263ba431662c60dd205de8bb0cf7485d4d12b3351" gracePeriod=30 Jul 08 19:01:08.118819 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:01:08.118780 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:01:17.148904 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:01:17.148865 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:01:28.095960 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:01:28.095922 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:01:30.196896 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:01:30.196863 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725620-z86z2" Jul 08 19:01:30.725028 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:01:30.724982 2570 generic.go:356] "Generic (PLEG): container finished" podID="2b41b382-a14c-4f73-a3f6-59cf83900ad3" containerID="83fb6231506f8d27329a8c3263ba431662c60dd205de8bb0cf7485d4d12b3351" exitCode=137 Jul 08 19:01:30.725247 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:01:30.725038 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725620-z86z2" Jul 08 19:01:30.725247 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:01:30.725065 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725620-z86z2" event={"ID":"2b41b382-a14c-4f73-a3f6-59cf83900ad3","Type":"ContainerDied","Data":"83fb6231506f8d27329a8c3263ba431662c60dd205de8bb0cf7485d4d12b3351"} Jul 08 19:01:30.725247 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:01:30.725120 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725620-z86z2" event={"ID":"2b41b382-a14c-4f73-a3f6-59cf83900ad3","Type":"ContainerDied","Data":"04be29473376b07f6cda4913fe823e29c9de729db42c491635d31e7658d07c9e"} Jul 08 19:01:30.725247 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:01:30.725144 2570 scope.go:117] "RemoveContainer" containerID="83fb6231506f8d27329a8c3263ba431662c60dd205de8bb0cf7485d4d12b3351" Jul 08 19:01:30.734966 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:01:30.734945 2570 scope.go:117] "RemoveContainer" containerID="83fb6231506f8d27329a8c3263ba431662c60dd205de8bb0cf7485d4d12b3351" Jul 08 19:01:30.735287 ip-10-0-130-48 kubenswrapper[2570]: E0708 19:01:30.735248 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83fb6231506f8d27329a8c3263ba431662c60dd205de8bb0cf7485d4d12b3351\": container with ID starting with 83fb6231506f8d27329a8c3263ba431662c60dd205de8bb0cf7485d4d12b3351 not found: ID does not exist" containerID="83fb6231506f8d27329a8c3263ba431662c60dd205de8bb0cf7485d4d12b3351" Jul 08 19:01:30.735341 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:01:30.735300 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83fb6231506f8d27329a8c3263ba431662c60dd205de8bb0cf7485d4d12b3351"} err="failed to get container status \"83fb6231506f8d27329a8c3263ba431662c60dd205de8bb0cf7485d4d12b3351\": rpc error: code = NotFound desc = could not find container \"83fb6231506f8d27329a8c3263ba431662c60dd205de8bb0cf7485d4d12b3351\": container with ID starting with 83fb6231506f8d27329a8c3263ba431662c60dd205de8bb0cf7485d4d12b3351 not found: ID does not exist" Jul 08 19:01:30.747498 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:01:30.747463 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725620-z86z2"] Jul 08 19:01:30.751358 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:01:30.751328 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725620-z86z2"] Jul 08 19:01:31.690242 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:01:31.690209 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b41b382-a14c-4f73-a3f6-59cf83900ad3" path="/var/lib/kubelet/pods/2b41b382-a14c-4f73-a3f6-59cf83900ad3/volumes" Jul 08 19:01:39.197913 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:01:39.197873 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:01:48.736068 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:01:48.736030 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:02:01.302377 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:02:01.302283 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:02:11.211611 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:02:11.211563 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:02:32.926425 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:02:32.926390 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:02:45.999201 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:02:45.999159 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:03:09.296767 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:03:09.296730 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:03:19.597470 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:03:19.597435 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:03:36.132497 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:03:36.132381 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:03:46.199746 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:03:46.199703 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:03:56.692526 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:03:56.692484 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:04:06.498006 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:04:06.497971 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:04:16.895270 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:04:16.895218 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:04:41.968236 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:04:41.968193 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tkzsc"] Jul 08 19:04:41.968732 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:04:41.968411 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tkzsc" podUID="832b207d-e75f-40b9-89b4-79421084796a" containerName="manager" containerID="cri-o://fa77a10a865efbac98000125c462f764c9be787379e01718b26117ce6c3b1563" gracePeriod=10 Jul 08 19:04:44.821893 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:04:44.821861 2570 generic.go:356] "Generic (PLEG): container finished" podID="832b207d-e75f-40b9-89b4-79421084796a" containerID="fa77a10a865efbac98000125c462f764c9be787379e01718b26117ce6c3b1563" exitCode=0 Jul 08 19:04:44.822221 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:04:44.821911 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tkzsc" event={"ID":"832b207d-e75f-40b9-89b4-79421084796a","Type":"ContainerDied","Data":"fa77a10a865efbac98000125c462f764c9be787379e01718b26117ce6c3b1563"} Jul 08 19:04:44.822221 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:04:44.821979 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tkzsc" event={"ID":"832b207d-e75f-40b9-89b4-79421084796a","Type":"ContainerDied","Data":"5ed79674a746f063099f7b6177d69df4052354b8fec070b77b9ef7cfc9df5b16"} Jul 08 19:04:44.822221 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:04:44.822023 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ed79674a746f063099f7b6177d69df4052354b8fec070b77b9ef7cfc9df5b16" Jul 08 19:04:44.830024 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:04:44.830006 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tkzsc" Jul 08 19:04:44.967107 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:04:44.967029 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/832b207d-e75f-40b9-89b4-79421084796a-extensions-socket-volume\") pod \"832b207d-e75f-40b9-89b4-79421084796a\" (UID: \"832b207d-e75f-40b9-89b4-79421084796a\") " Jul 08 19:04:44.967250 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:04:44.967109 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kg876\" (UniqueName: \"kubernetes.io/projected/832b207d-e75f-40b9-89b4-79421084796a-kube-api-access-kg876\") pod \"832b207d-e75f-40b9-89b4-79421084796a\" (UID: \"832b207d-e75f-40b9-89b4-79421084796a\") " Jul 08 19:04:44.967408 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:04:44.967386 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832b207d-e75f-40b9-89b4-79421084796a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "832b207d-e75f-40b9-89b4-79421084796a" (UID: "832b207d-e75f-40b9-89b4-79421084796a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:04:44.969026 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:04:44.969007 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/832b207d-e75f-40b9-89b4-79421084796a-kube-api-access-kg876" (OuterVolumeSpecName: "kube-api-access-kg876") pod "832b207d-e75f-40b9-89b4-79421084796a" (UID: "832b207d-e75f-40b9-89b4-79421084796a"). InnerVolumeSpecName "kube-api-access-kg876". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:04:45.068190 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:04:45.068155 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg876\" (UniqueName: \"kubernetes.io/projected/832b207d-e75f-40b9-89b4-79421084796a-kube-api-access-kg876\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:04:45.068190 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:04:45.068183 2570 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/832b207d-e75f-40b9-89b4-79421084796a-extensions-socket-volume\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:04:45.827111 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:04:45.827085 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tkzsc" Jul 08 19:04:45.846537 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:04:45.846510 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tkzsc"] Jul 08 19:04:45.851380 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:04:45.851358 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tkzsc"] Jul 08 19:04:47.690074 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:04:47.690042 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="832b207d-e75f-40b9-89b4-79421084796a" path="/var/lib/kubelet/pods/832b207d-e75f-40b9-89b4-79421084796a/volumes" Jul 08 19:05:39.703280 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:05:39.703204 2570 scope.go:117] "RemoveContainer" containerID="fa77a10a865efbac98000125c462f764c9be787379e01718b26117ce6c3b1563" Jul 08 19:05:43.132134 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:05:43.132096 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dh7sh"] Jul 08 19:05:43.132571 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:05:43.132557 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b41b382-a14c-4f73-a3f6-59cf83900ad3" containerName="cleanup" Jul 08 19:05:43.132616 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:05:43.132573 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b41b382-a14c-4f73-a3f6-59cf83900ad3" containerName="cleanup" Jul 08 19:05:43.132616 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:05:43.132592 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="832b207d-e75f-40b9-89b4-79421084796a" containerName="manager" Jul 08 19:05:43.132616 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:05:43.132598 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="832b207d-e75f-40b9-89b4-79421084796a" containerName="manager" Jul 08 19:05:43.132751 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:05:43.132657 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="832b207d-e75f-40b9-89b4-79421084796a" containerName="manager" Jul 08 19:05:43.132751 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:05:43.132666 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b41b382-a14c-4f73-a3f6-59cf83900ad3" containerName="cleanup" Jul 08 19:05:43.137392 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:05:43.137367 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dh7sh" Jul 08 19:05:43.139983 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:05:43.139963 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rxz88\"" Jul 08 19:05:43.149418 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:05:43.149395 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dh7sh"] Jul 08 19:05:43.172609 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:05:43.172580 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4183c419-1826-4851-9952-d26a07ed80aa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dh7sh\" (UID: \"4183c419-1826-4851-9952-d26a07ed80aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dh7sh" Jul 08 19:05:43.172772 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:05:43.172622 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtnp6\" (UniqueName: \"kubernetes.io/projected/4183c419-1826-4851-9952-d26a07ed80aa-kube-api-access-xtnp6\") pod \"kuadrant-operator-controller-manager-86878c769f-dh7sh\" (UID: \"4183c419-1826-4851-9952-d26a07ed80aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dh7sh" Jul 08 19:05:43.273419 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:05:43.273379 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4183c419-1826-4851-9952-d26a07ed80aa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dh7sh\" (UID: \"4183c419-1826-4851-9952-d26a07ed80aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dh7sh" Jul 08 19:05:43.273619 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:05:43.273430 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtnp6\" (UniqueName: \"kubernetes.io/projected/4183c419-1826-4851-9952-d26a07ed80aa-kube-api-access-xtnp6\") pod \"kuadrant-operator-controller-manager-86878c769f-dh7sh\" (UID: \"4183c419-1826-4851-9952-d26a07ed80aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dh7sh" Jul 08 19:05:43.273790 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:05:43.273771 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4183c419-1826-4851-9952-d26a07ed80aa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dh7sh\" (UID: \"4183c419-1826-4851-9952-d26a07ed80aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dh7sh" Jul 08 19:05:43.286166 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:05:43.286143 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtnp6\" (UniqueName: \"kubernetes.io/projected/4183c419-1826-4851-9952-d26a07ed80aa-kube-api-access-xtnp6\") pod \"kuadrant-operator-controller-manager-86878c769f-dh7sh\" (UID: \"4183c419-1826-4851-9952-d26a07ed80aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dh7sh" Jul 08 19:05:43.448457 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:05:43.448361 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dh7sh" Jul 08 19:05:43.597800 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:05:43.597700 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dh7sh"] Jul 08 19:05:43.601075 ip-10-0-130-48 kubenswrapper[2570]: W0708 19:05:43.601038 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4183c419_1826_4851_9952_d26a07ed80aa.slice/crio-28a21d095a96139e1e9ee0a9eb69480f653dbeb4a1c52dd1127128793b83c86d WatchSource:0}: Error finding container 28a21d095a96139e1e9ee0a9eb69480f653dbeb4a1c52dd1127128793b83c86d: Status 404 returned error can't find the container with id 28a21d095a96139e1e9ee0a9eb69480f653dbeb4a1c52dd1127128793b83c86d Jul 08 19:05:44.158134 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:05:44.158094 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dh7sh" event={"ID":"4183c419-1826-4851-9952-d26a07ed80aa","Type":"ContainerStarted","Data":"b02c6d168ae2ef61a9309493782f3a2c8026e05b50a3f5e6e0da0726ee360184"} Jul 08 19:05:44.158591 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:05:44.158161 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dh7sh" event={"ID":"4183c419-1826-4851-9952-d26a07ed80aa","Type":"ContainerStarted","Data":"28a21d095a96139e1e9ee0a9eb69480f653dbeb4a1c52dd1127128793b83c86d"} Jul 08 19:05:44.158591 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:05:44.158231 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dh7sh" Jul 08 19:05:44.181427 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:05:44.181381 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dh7sh" podStartSLOduration=1.181366226 podStartE2EDuration="1.181366226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:05:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:05:44.179011391 +0000 UTC m=+1205.026286060" watchObservedRunningTime="2026-07-08 19:05:44.181366226 +0000 UTC m=+1205.028640892" Jul 08 19:05:55.165764 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:05:55.165734 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dh7sh" Jul 08 19:06:02.894718 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:06:02.894675 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:06:15.002873 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:06:15.002840 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:06:23.303502 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:06:23.303465 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:06:35.026247 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:06:35.026165 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:06:45.486835 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:06:45.486790 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:06:56.493325 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:06:56.493287 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:07:04.795698 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:07:04.795662 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:07:15.993557 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:07:15.993519 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:07:26.311483 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:07:26.311450 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:07:36.789387 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:07:36.789352 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:07:50.298363 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:07:50.298324 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:08:11.813174 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:08:11.813087 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:08:19.309735 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:08:19.309695 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:08:29.089411 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:08:29.089380 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:08:38.794646 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:08:38.794609 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:08:45.293124 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:08:45.293082 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:09:01.803985 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:09:01.803952 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:09:10.901132 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:09:10.901095 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:09:20.211444 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:09:20.211404 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:09:27.990616 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:09:27.990579 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:09:38.530724 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:09:38.530645 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:09:48.995240 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:09:48.995208 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:10:02.423301 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:10:02.423264 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:10:11.859664 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:10:11.859628 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:10:21.006092 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:10:21.006052 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:10:31.290712 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:10:31.290677 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:10:43.297289 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:10:43.297238 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:10:52.702336 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:10:52.702295 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:11:04.697721 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:11:04.697638 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:11:15.803679 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:11:15.803647 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:11:28.805335 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:11:28.805298 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:11:37.914550 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:11:37.914510 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:11:50.334947 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:11:50.334910 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:11:59.373566 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:11:59.373532 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:12:07.312883 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:07.312848 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92"] Jul 08 19:12:07.316836 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:07.316808 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" Jul 08 19:12:07.321589 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:07.321565 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-787671dd-openshift-default-dockercfg-7kpkt\"" Jul 08 19:12:07.422377 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:07.422289 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/144ac354-e7eb-4a6b-97f2-e8d52de2626f-workload-certs\") pod \"e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92\" (UID: \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\") " pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" Jul 08 19:12:07.422377 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:07.422328 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvgtz\" (UniqueName: \"kubernetes.io/projected/144ac354-e7eb-4a6b-97f2-e8d52de2626f-kube-api-access-lvgtz\") pod \"e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92\" (UID: \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\") " pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" Jul 08 19:12:07.422377 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:07.422352 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/144ac354-e7eb-4a6b-97f2-e8d52de2626f-workload-socket\") pod \"e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92\" (UID: \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\") " pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" Jul 08 19:12:07.422609 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:07.422446 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/144ac354-e7eb-4a6b-97f2-e8d52de2626f-istio-envoy\") pod \"e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92\" (UID: \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\") " pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" Jul 08 19:12:07.422609 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:07.422478 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/144ac354-e7eb-4a6b-97f2-e8d52de2626f-istio-podinfo\") pod \"e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92\" (UID: \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\") " pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" Jul 08 19:12:07.422609 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:07.422510 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/144ac354-e7eb-4a6b-97f2-e8d52de2626f-istio-data\") pod \"e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92\" (UID: \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\") " pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" Jul 08 19:12:07.422609 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:07.422570 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/144ac354-e7eb-4a6b-97f2-e8d52de2626f-istio-token\") pod \"e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92\" (UID: \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\") " pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" Jul 08 19:12:07.422609 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:07.422605 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/144ac354-e7eb-4a6b-97f2-e8d52de2626f-istiod-ca-cert\") pod \"e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92\" (UID: \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\") " pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" Jul 08 19:12:07.422769 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:07.422667 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/144ac354-e7eb-4a6b-97f2-e8d52de2626f-credential-socket\") pod \"e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92\" (UID: \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\") " pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" Jul 08 19:12:07.455714 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:07.455679 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92"] Jul 08 19:12:07.523103 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:07.523075 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/144ac354-e7eb-4a6b-97f2-e8d52de2626f-istio-data\") pod \"e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92\" (UID: \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\") " pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" Jul 08 19:12:07.523292 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:07.523109 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/144ac354-e7eb-4a6b-97f2-e8d52de2626f-istio-token\") pod \"e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92\" (UID: \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\") " pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" Jul 08 19:12:07.523292 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:07.523142 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/144ac354-e7eb-4a6b-97f2-e8d52de2626f-istiod-ca-cert\") pod \"e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92\" (UID: \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\") " pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" Jul 08 19:12:07.523292 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:07.523198 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/144ac354-e7eb-4a6b-97f2-e8d52de2626f-credential-socket\") pod \"e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92\" (UID: \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\") " pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" Jul 08 19:12:07.523440 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:07.523292 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/144ac354-e7eb-4a6b-97f2-e8d52de2626f-workload-certs\") pod \"e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92\" (UID: \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\") " pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" Jul 08 19:12:07.523440 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:07.523326 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvgtz\" (UniqueName: \"kubernetes.io/projected/144ac354-e7eb-4a6b-97f2-e8d52de2626f-kube-api-access-lvgtz\") pod \"e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92\" (UID: \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\") " pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" Jul 08 19:12:07.523440 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:07.523361 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/144ac354-e7eb-4a6b-97f2-e8d52de2626f-workload-socket\") pod \"e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92\" (UID: \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\") " pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" Jul 08 19:12:07.523440 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:07.523420 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/144ac354-e7eb-4a6b-97f2-e8d52de2626f-istio-envoy\") pod \"e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92\" (UID: \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\") " pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" Jul 08 19:12:07.523633 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:07.523452 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/144ac354-e7eb-4a6b-97f2-e8d52de2626f-istio-podinfo\") pod \"e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92\" (UID: \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\") " pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" Jul 08 19:12:07.523633 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:07.523536 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/144ac354-e7eb-4a6b-97f2-e8d52de2626f-istio-data\") pod \"e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92\" (UID: \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\") " pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" Jul 08 19:12:07.523739 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:07.523694 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/144ac354-e7eb-4a6b-97f2-e8d52de2626f-credential-socket\") pod \"e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92\" (UID: \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\") " pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" Jul 08 19:12:07.523958 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:07.523907 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/144ac354-e7eb-4a6b-97f2-e8d52de2626f-workload-certs\") pod \"e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92\" (UID: \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\") " pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" Jul 08 19:12:07.523958 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:07.523940 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/144ac354-e7eb-4a6b-97f2-e8d52de2626f-workload-socket\") pod \"e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92\" (UID: \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\") " pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" Jul 08 19:12:07.524105 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:07.524016 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/144ac354-e7eb-4a6b-97f2-e8d52de2626f-istiod-ca-cert\") pod \"e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92\" (UID: \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\") " pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" Jul 08 19:12:07.525764 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:07.525737 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/144ac354-e7eb-4a6b-97f2-e8d52de2626f-istio-envoy\") pod \"e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92\" (UID: \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\") " pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" Jul 08 19:12:07.525990 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:07.525973 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/144ac354-e7eb-4a6b-97f2-e8d52de2626f-istio-podinfo\") pod \"e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92\" (UID: \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\") " pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" Jul 08 19:12:07.548702 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:07.548677 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/144ac354-e7eb-4a6b-97f2-e8d52de2626f-istio-token\") pod \"e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92\" (UID: \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\") " pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" Jul 08 19:12:07.549318 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:07.549296 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvgtz\" (UniqueName: \"kubernetes.io/projected/144ac354-e7eb-4a6b-97f2-e8d52de2626f-kube-api-access-lvgtz\") pod \"e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92\" (UID: \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\") " pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" Jul 08 19:12:07.628205 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:07.628168 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" Jul 08 19:12:07.826313 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:07.826280 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92"] Jul 08 19:12:07.829165 ip-10-0-130-48 kubenswrapper[2570]: W0708 19:12:07.829133 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod144ac354_e7eb_4a6b_97f2_e8d52de2626f.slice/crio-8ba7d80c567ffe1a65828fce1fd475d2ad1fbc40a09b3d23a69153be5e9b5355 WatchSource:0}: Error finding container 8ba7d80c567ffe1a65828fce1fd475d2ad1fbc40a09b3d23a69153be5e9b5355: Status 404 returned error can't find the container with id 8ba7d80c567ffe1a65828fce1fd475d2ad1fbc40a09b3d23a69153be5e9b5355 Jul 08 19:12:07.831830 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:07.831790 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:12:07.831942 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:07.831880 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:12:07.831942 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:07.831923 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:12:08.330721 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:08.330683 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" event={"ID":"144ac354-e7eb-4a6b-97f2-e8d52de2626f","Type":"ContainerStarted","Data":"c707fa62166c3d60bec4ea4910ca2339b259bfe8388ade938e2b7e2451b4d7fb"} Jul 08 19:12:08.331122 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:08.330750 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" event={"ID":"144ac354-e7eb-4a6b-97f2-e8d52de2626f","Type":"ContainerStarted","Data":"8ba7d80c567ffe1a65828fce1fd475d2ad1fbc40a09b3d23a69153be5e9b5355"} Jul 08 19:12:08.389026 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:08.388975 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" podStartSLOduration=1.388957671 podStartE2EDuration="1.388957671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:12:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:12:08.384092237 +0000 UTC m=+1589.231366902" watchObservedRunningTime="2026-07-08 19:12:08.388957671 +0000 UTC m=+1589.236232330" Jul 08 19:12:08.551049 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:08.550984 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-787671dd-68c5c8796-bzvpz"] Jul 08 19:12:08.554776 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:08.554754 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-787671dd-68c5c8796-bzvpz" Jul 08 19:12:08.558146 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:08.558082 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-787671dd-serving-cert\"" Jul 08 19:12:08.558293 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:08.558220 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s5q9r\"" Jul 08 19:12:08.583754 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:08.583691 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-787671dd-68c5c8796-bzvpz"] Jul 08 19:12:08.628322 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:08.628292 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" Jul 08 19:12:08.634303 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:08.634271 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" Jul 08 19:12:08.637646 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:08.637622 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b14e2868-92b4-433b-9f87-b19788d1757c-openshift-service-ca\") pod \"maas-api-e2e-ait-787671dd-68c5c8796-bzvpz\" (UID: \"b14e2868-92b4-433b-9f87-b19788d1757c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-787671dd-68c5c8796-bzvpz" Jul 08 19:12:08.637780 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:08.637669 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b14e2868-92b4-433b-9f87-b19788d1757c-maas-api-tls\") pod \"maas-api-e2e-ait-787671dd-68c5c8796-bzvpz\" (UID: \"b14e2868-92b4-433b-9f87-b19788d1757c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-787671dd-68c5c8796-bzvpz" Jul 08 19:12:08.637780 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:08.637768 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5flz5\" (UniqueName: \"kubernetes.io/projected/b14e2868-92b4-433b-9f87-b19788d1757c-kube-api-access-5flz5\") pod \"maas-api-e2e-ait-787671dd-68c5c8796-bzvpz\" (UID: \"b14e2868-92b4-433b-9f87-b19788d1757c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-787671dd-68c5c8796-bzvpz" Jul 08 19:12:08.739110 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:08.739073 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b14e2868-92b4-433b-9f87-b19788d1757c-openshift-service-ca\") pod \"maas-api-e2e-ait-787671dd-68c5c8796-bzvpz\" (UID: \"b14e2868-92b4-433b-9f87-b19788d1757c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-787671dd-68c5c8796-bzvpz" Jul 08 19:12:08.739323 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:08.739130 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b14e2868-92b4-433b-9f87-b19788d1757c-maas-api-tls\") pod \"maas-api-e2e-ait-787671dd-68c5c8796-bzvpz\" (UID: \"b14e2868-92b4-433b-9f87-b19788d1757c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-787671dd-68c5c8796-bzvpz" Jul 08 19:12:08.739323 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:08.739214 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5flz5\" (UniqueName: \"kubernetes.io/projected/b14e2868-92b4-433b-9f87-b19788d1757c-kube-api-access-5flz5\") pod \"maas-api-e2e-ait-787671dd-68c5c8796-bzvpz\" (UID: \"b14e2868-92b4-433b-9f87-b19788d1757c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-787671dd-68c5c8796-bzvpz" Jul 08 19:12:08.739323 ip-10-0-130-48 kubenswrapper[2570]: E0708 19:12:08.739234 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-787671dd-serving-cert: secret "maas-api-e2e-ait-787671dd-serving-cert" not found Jul 08 19:12:08.739514 ip-10-0-130-48 kubenswrapper[2570]: E0708 19:12:08.739321 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b14e2868-92b4-433b-9f87-b19788d1757c-maas-api-tls podName:b14e2868-92b4-433b-9f87-b19788d1757c nodeName:}" failed. No retries permitted until 2026-07-08 19:12:09.239301591 +0000 UTC m=+1590.086576241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b14e2868-92b4-433b-9f87-b19788d1757c-maas-api-tls") pod "maas-api-e2e-ait-787671dd-68c5c8796-bzvpz" (UID: "b14e2868-92b4-433b-9f87-b19788d1757c") : secret "maas-api-e2e-ait-787671dd-serving-cert" not found Jul 08 19:12:08.739846 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:08.739827 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b14e2868-92b4-433b-9f87-b19788d1757c-openshift-service-ca\") pod \"maas-api-e2e-ait-787671dd-68c5c8796-bzvpz\" (UID: \"b14e2868-92b4-433b-9f87-b19788d1757c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-787671dd-68c5c8796-bzvpz" Jul 08 19:12:08.753858 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:08.753790 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5flz5\" (UniqueName: \"kubernetes.io/projected/b14e2868-92b4-433b-9f87-b19788d1757c-kube-api-access-5flz5\") pod \"maas-api-e2e-ait-787671dd-68c5c8796-bzvpz\" (UID: \"b14e2868-92b4-433b-9f87-b19788d1757c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-787671dd-68c5c8796-bzvpz" Jul 08 19:12:09.245896 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:09.245860 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b14e2868-92b4-433b-9f87-b19788d1757c-maas-api-tls\") pod \"maas-api-e2e-ait-787671dd-68c5c8796-bzvpz\" (UID: \"b14e2868-92b4-433b-9f87-b19788d1757c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-787671dd-68c5c8796-bzvpz" Jul 08 19:12:09.248482 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:09.248452 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b14e2868-92b4-433b-9f87-b19788d1757c-maas-api-tls\") pod \"maas-api-e2e-ait-787671dd-68c5c8796-bzvpz\" (UID: \"b14e2868-92b4-433b-9f87-b19788d1757c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-787671dd-68c5c8796-bzvpz" Jul 08 19:12:09.336720 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:09.336686 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" Jul 08 19:12:09.337960 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:09.337938 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" Jul 08 19:12:09.467074 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:09.467046 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-787671dd-68c5c8796-bzvpz" Jul 08 19:12:09.628561 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:09.628527 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-787671dd-68c5c8796-bzvpz"] Jul 08 19:12:09.633753 ip-10-0-130-48 kubenswrapper[2570]: W0708 19:12:09.633715 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb14e2868_92b4_433b_9f87_b19788d1757c.slice/crio-62ce59e49ca0ad1e86ea42a4e8ed0a6520b63e8b5358ada1292b228753f2847f WatchSource:0}: Error finding container 62ce59e49ca0ad1e86ea42a4e8ed0a6520b63e8b5358ada1292b228753f2847f: Status 404 returned error can't find the container with id 62ce59e49ca0ad1e86ea42a4e8ed0a6520b63e8b5358ada1292b228753f2847f Jul 08 19:12:09.635371 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:09.635350 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 19:12:10.344531 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:10.344496 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-787671dd-68c5c8796-bzvpz" event={"ID":"b14e2868-92b4-433b-9f87-b19788d1757c","Type":"ContainerStarted","Data":"62ce59e49ca0ad1e86ea42a4e8ed0a6520b63e8b5358ada1292b228753f2847f"} Jul 08 19:12:11.353519 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:11.353483 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-787671dd-68c5c8796-bzvpz" event={"ID":"b14e2868-92b4-433b-9f87-b19788d1757c","Type":"ContainerStarted","Data":"5c8aae9099d86b397829d481aa4b8d1e403c0b9d3ec760b05743579cf3db0496"} Jul 08 19:12:11.353953 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:11.353734 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-787671dd-68c5c8796-bzvpz" Jul 08 19:12:12.724387 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:12.724328 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-787671dd-68c5c8796-bzvpz" podStartSLOduration=4.006216393 podStartE2EDuration="4.724307986s" totalImagesPullingTime="718.091593ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:12:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:12:11.41308747 +0000 UTC m=+1592.260362136" watchObservedRunningTime="2026-07-08 19:12:12.724307986 +0000 UTC m=+1593.571582651" Jul 08 19:12:12.725341 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:12.725322 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:12:13.857420 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:13.857375 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92"] Jul 08 19:12:13.857897 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:13.857620 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" podUID="144ac354-e7eb-4a6b-97f2-e8d52de2626f" containerName="istio-proxy" containerID="cri-o://c707fa62166c3d60bec4ea4910ca2339b259bfe8388ade938e2b7e2451b4d7fb" gracePeriod=30 Jul 08 19:12:14.578562 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:14.578529 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-787671dd-68c5c8796-bzvpz"] Jul 08 19:12:14.578748 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:14.578707 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-787671dd-68c5c8796-bzvpz" podUID="b14e2868-92b4-433b-9f87-b19788d1757c" containerName="maas-api" containerID="cri-o://5c8aae9099d86b397829d481aa4b8d1e403c0b9d3ec760b05743579cf3db0496" gracePeriod=30 Jul 08 19:12:14.883343 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:14.883319 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-787671dd-68c5c8796-bzvpz" Jul 08 19:12:14.998345 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:14.998307 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5flz5\" (UniqueName: \"kubernetes.io/projected/b14e2868-92b4-433b-9f87-b19788d1757c-kube-api-access-5flz5\") pod \"b14e2868-92b4-433b-9f87-b19788d1757c\" (UID: \"b14e2868-92b4-433b-9f87-b19788d1757c\") " Jul 08 19:12:14.998564 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:14.998398 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b14e2868-92b4-433b-9f87-b19788d1757c-openshift-service-ca\") pod \"b14e2868-92b4-433b-9f87-b19788d1757c\" (UID: \"b14e2868-92b4-433b-9f87-b19788d1757c\") " Jul 08 19:12:14.998648 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:14.998561 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b14e2868-92b4-433b-9f87-b19788d1757c-maas-api-tls\") pod \"b14e2868-92b4-433b-9f87-b19788d1757c\" (UID: \"b14e2868-92b4-433b-9f87-b19788d1757c\") " Jul 08 19:12:14.998732 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:14.998701 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b14e2868-92b4-433b-9f87-b19788d1757c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b14e2868-92b4-433b-9f87-b19788d1757c" (UID: "b14e2868-92b4-433b-9f87-b19788d1757c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:12:14.998951 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:14.998931 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b14e2868-92b4-433b-9f87-b19788d1757c-openshift-service-ca\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:12:15.000407 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:15.000376 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b14e2868-92b4-433b-9f87-b19788d1757c-kube-api-access-5flz5" (OuterVolumeSpecName: "kube-api-access-5flz5") pod "b14e2868-92b4-433b-9f87-b19788d1757c" (UID: "b14e2868-92b4-433b-9f87-b19788d1757c"). InnerVolumeSpecName "kube-api-access-5flz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:12:15.000624 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:15.000603 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b14e2868-92b4-433b-9f87-b19788d1757c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b14e2868-92b4-433b-9f87-b19788d1757c" (UID: "b14e2868-92b4-433b-9f87-b19788d1757c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:12:15.099546 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:15.099515 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b14e2868-92b4-433b-9f87-b19788d1757c-maas-api-tls\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:12:15.099546 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:15.099544 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5flz5\" (UniqueName: \"kubernetes.io/projected/b14e2868-92b4-433b-9f87-b19788d1757c-kube-api-access-5flz5\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:12:15.378052 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:15.378016 2570 generic.go:356] "Generic (PLEG): container finished" podID="b14e2868-92b4-433b-9f87-b19788d1757c" containerID="5c8aae9099d86b397829d481aa4b8d1e403c0b9d3ec760b05743579cf3db0496" exitCode=0 Jul 08 19:12:15.378248 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:15.378070 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-787671dd-68c5c8796-bzvpz" Jul 08 19:12:15.378248 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:15.378112 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-787671dd-68c5c8796-bzvpz" event={"ID":"b14e2868-92b4-433b-9f87-b19788d1757c","Type":"ContainerDied","Data":"5c8aae9099d86b397829d481aa4b8d1e403c0b9d3ec760b05743579cf3db0496"} Jul 08 19:12:15.378248 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:15.378206 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-787671dd-68c5c8796-bzvpz" event={"ID":"b14e2868-92b4-433b-9f87-b19788d1757c","Type":"ContainerDied","Data":"62ce59e49ca0ad1e86ea42a4e8ed0a6520b63e8b5358ada1292b228753f2847f"} Jul 08 19:12:15.378248 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:15.378233 2570 scope.go:117] "RemoveContainer" containerID="5c8aae9099d86b397829d481aa4b8d1e403c0b9d3ec760b05743579cf3db0496" Jul 08 19:12:15.387943 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:15.387924 2570 scope.go:117] "RemoveContainer" containerID="5c8aae9099d86b397829d481aa4b8d1e403c0b9d3ec760b05743579cf3db0496" Jul 08 19:12:15.388219 ip-10-0-130-48 kubenswrapper[2570]: E0708 19:12:15.388199 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c8aae9099d86b397829d481aa4b8d1e403c0b9d3ec760b05743579cf3db0496\": container with ID starting with 5c8aae9099d86b397829d481aa4b8d1e403c0b9d3ec760b05743579cf3db0496 not found: ID does not exist" containerID="5c8aae9099d86b397829d481aa4b8d1e403c0b9d3ec760b05743579cf3db0496" Jul 08 19:12:15.388313 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:15.388227 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c8aae9099d86b397829d481aa4b8d1e403c0b9d3ec760b05743579cf3db0496"} err="failed to get container status \"5c8aae9099d86b397829d481aa4b8d1e403c0b9d3ec760b05743579cf3db0496\": rpc error: code = NotFound desc = could not find container \"5c8aae9099d86b397829d481aa4b8d1e403c0b9d3ec760b05743579cf3db0496\": container with ID starting with 5c8aae9099d86b397829d481aa4b8d1e403c0b9d3ec760b05743579cf3db0496 not found: ID does not exist" Jul 08 19:12:15.433247 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:15.433211 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-787671dd-68c5c8796-bzvpz"] Jul 08 19:12:15.445106 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:15.445059 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-787671dd-68c5c8796-bzvpz"] Jul 08 19:12:15.690396 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:15.690315 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b14e2868-92b4-433b-9f87-b19788d1757c" path="/var/lib/kubelet/pods/b14e2868-92b4-433b-9f87-b19788d1757c/volumes" Jul 08 19:12:20.410393 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:20.410366 2570 generic.go:356] "Generic (PLEG): container finished" podID="144ac354-e7eb-4a6b-97f2-e8d52de2626f" containerID="c707fa62166c3d60bec4ea4910ca2339b259bfe8388ade938e2b7e2451b4d7fb" exitCode=0 Jul 08 19:12:20.410780 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:20.410438 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" event={"ID":"144ac354-e7eb-4a6b-97f2-e8d52de2626f","Type":"ContainerDied","Data":"c707fa62166c3d60bec4ea4910ca2339b259bfe8388ade938e2b7e2451b4d7fb"} Jul 08 19:12:20.410780 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:20.410505 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" event={"ID":"144ac354-e7eb-4a6b-97f2-e8d52de2626f","Type":"ContainerDied","Data":"8ba7d80c567ffe1a65828fce1fd475d2ad1fbc40a09b3d23a69153be5e9b5355"} Jul 08 19:12:20.410780 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:20.410534 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ba7d80c567ffe1a65828fce1fd475d2ad1fbc40a09b3d23a69153be5e9b5355" Jul 08 19:12:20.416490 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:20.416445 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" Jul 08 19:12:20.554616 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:20.554538 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/144ac354-e7eb-4a6b-97f2-e8d52de2626f-istio-data\") pod \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\" (UID: \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\") " Jul 08 19:12:20.554616 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:20.554579 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvgtz\" (UniqueName: \"kubernetes.io/projected/144ac354-e7eb-4a6b-97f2-e8d52de2626f-kube-api-access-lvgtz\") pod \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\" (UID: \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\") " Jul 08 19:12:20.554616 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:20.554612 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/144ac354-e7eb-4a6b-97f2-e8d52de2626f-workload-socket\") pod \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\" (UID: \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\") " Jul 08 19:12:20.554881 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:20.554643 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/144ac354-e7eb-4a6b-97f2-e8d52de2626f-istio-podinfo\") pod \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\" (UID: \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\") " Jul 08 19:12:20.554881 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:20.554678 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/144ac354-e7eb-4a6b-97f2-e8d52de2626f-workload-certs\") pod \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\" (UID: \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\") " Jul 08 19:12:20.554881 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:20.554755 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/144ac354-e7eb-4a6b-97f2-e8d52de2626f-credential-socket\") pod \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\" (UID: \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\") " Jul 08 19:12:20.554881 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:20.554798 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/144ac354-e7eb-4a6b-97f2-e8d52de2626f-istio-token\") pod \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\" (UID: \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\") " Jul 08 19:12:20.554881 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:20.554861 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/144ac354-e7eb-4a6b-97f2-e8d52de2626f-istiod-ca-cert\") pod \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\" (UID: \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\") " Jul 08 19:12:20.555130 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:20.554907 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/144ac354-e7eb-4a6b-97f2-e8d52de2626f-istio-envoy\") pod \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\" (UID: \"144ac354-e7eb-4a6b-97f2-e8d52de2626f\") " Jul 08 19:12:20.555130 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:20.554955 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144ac354-e7eb-4a6b-97f2-e8d52de2626f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "144ac354-e7eb-4a6b-97f2-e8d52de2626f" (UID: "144ac354-e7eb-4a6b-97f2-e8d52de2626f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:12:20.555130 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:20.555018 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144ac354-e7eb-4a6b-97f2-e8d52de2626f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "144ac354-e7eb-4a6b-97f2-e8d52de2626f" (UID: "144ac354-e7eb-4a6b-97f2-e8d52de2626f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:12:20.555314 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:20.555286 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/144ac354-e7eb-4a6b-97f2-e8d52de2626f-credential-socket\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:12:20.555314 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:20.555306 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/144ac354-e7eb-4a6b-97f2-e8d52de2626f-workload-socket\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:12:20.555314 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:20.555301 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144ac354-e7eb-4a6b-97f2-e8d52de2626f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "144ac354-e7eb-4a6b-97f2-e8d52de2626f" (UID: "144ac354-e7eb-4a6b-97f2-e8d52de2626f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:12:20.555492 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:20.555345 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144ac354-e7eb-4a6b-97f2-e8d52de2626f-istio-data" (OuterVolumeSpecName: "istio-data") pod "144ac354-e7eb-4a6b-97f2-e8d52de2626f" (UID: "144ac354-e7eb-4a6b-97f2-e8d52de2626f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:12:20.555492 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:20.555393 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/144ac354-e7eb-4a6b-97f2-e8d52de2626f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "144ac354-e7eb-4a6b-97f2-e8d52de2626f" (UID: "144ac354-e7eb-4a6b-97f2-e8d52de2626f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:12:20.556944 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:20.556912 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/144ac354-e7eb-4a6b-97f2-e8d52de2626f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "144ac354-e7eb-4a6b-97f2-e8d52de2626f" (UID: "144ac354-e7eb-4a6b-97f2-e8d52de2626f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 19:12:20.557056 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:20.556920 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144ac354-e7eb-4a6b-97f2-e8d52de2626f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "144ac354-e7eb-4a6b-97f2-e8d52de2626f" (UID: "144ac354-e7eb-4a6b-97f2-e8d52de2626f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:12:20.557056 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:20.557027 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/144ac354-e7eb-4a6b-97f2-e8d52de2626f-kube-api-access-lvgtz" (OuterVolumeSpecName: "kube-api-access-lvgtz") pod "144ac354-e7eb-4a6b-97f2-e8d52de2626f" (UID: "144ac354-e7eb-4a6b-97f2-e8d52de2626f"). InnerVolumeSpecName "kube-api-access-lvgtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:12:20.557056 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:20.557035 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/144ac354-e7eb-4a6b-97f2-e8d52de2626f-istio-token" (OuterVolumeSpecName: "istio-token") pod "144ac354-e7eb-4a6b-97f2-e8d52de2626f" (UID: "144ac354-e7eb-4a6b-97f2-e8d52de2626f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:12:20.656427 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:20.656388 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/144ac354-e7eb-4a6b-97f2-e8d52de2626f-istio-token\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:12:20.656427 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:20.656417 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/144ac354-e7eb-4a6b-97f2-e8d52de2626f-istiod-ca-cert\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:12:20.656427 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:20.656430 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/144ac354-e7eb-4a6b-97f2-e8d52de2626f-istio-envoy\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:12:20.656673 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:20.656441 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/144ac354-e7eb-4a6b-97f2-e8d52de2626f-istio-data\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:12:20.656673 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:20.656453 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvgtz\" (UniqueName: \"kubernetes.io/projected/144ac354-e7eb-4a6b-97f2-e8d52de2626f-kube-api-access-lvgtz\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:12:20.656673 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:20.656465 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/144ac354-e7eb-4a6b-97f2-e8d52de2626f-istio-podinfo\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:12:20.656673 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:20.656478 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/144ac354-e7eb-4a6b-97f2-e8d52de2626f-workload-certs\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:12:21.418301 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:21.418269 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92" Jul 08 19:12:21.465822 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:21.465793 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92"] Jul 08 19:12:21.471590 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:21.471562 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-787671dd-openshift-default-7fdd655ff5-xsd92"] Jul 08 19:12:21.690607 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:21.690530 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="144ac354-e7eb-4a6b-97f2-e8d52de2626f" path="/var/lib/kubelet/pods/144ac354-e7eb-4a6b-97f2-e8d52de2626f/volumes" Jul 08 19:12:22.543667 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.543634 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n"] Jul 08 19:12:22.544045 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.544032 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b14e2868-92b4-433b-9f87-b19788d1757c" containerName="maas-api" Jul 08 19:12:22.544045 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.544045 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b14e2868-92b4-433b-9f87-b19788d1757c" containerName="maas-api" Jul 08 19:12:22.544119 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.544067 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="144ac354-e7eb-4a6b-97f2-e8d52de2626f" containerName="istio-proxy" Jul 08 19:12:22.544119 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.544073 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="144ac354-e7eb-4a6b-97f2-e8d52de2626f" containerName="istio-proxy" Jul 08 19:12:22.544182 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.544130 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b14e2868-92b4-433b-9f87-b19788d1757c" containerName="maas-api" Jul 08 19:12:22.544182 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.544140 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="144ac354-e7eb-4a6b-97f2-e8d52de2626f" containerName="istio-proxy" Jul 08 19:12:22.546760 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.546743 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" Jul 08 19:12:22.549907 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.549877 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-036c61dd-openshift-default-dockercfg-ncq5g\"" Jul 08 19:12:22.574334 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.574308 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a54fd058-ecef-4fc0-8b95-5b24396b78d1-istio-token\") pod \"e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n\" (UID: \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\") " pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" Jul 08 19:12:22.574471 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.574361 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a54fd058-ecef-4fc0-8b95-5b24396b78d1-istiod-ca-cert\") pod \"e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n\" (UID: \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\") " pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" Jul 08 19:12:22.574544 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.574471 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a54fd058-ecef-4fc0-8b95-5b24396b78d1-credential-socket\") pod \"e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n\" (UID: \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\") " pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" Jul 08 19:12:22.574544 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.574502 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a54fd058-ecef-4fc0-8b95-5b24396b78d1-istio-envoy\") pod \"e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n\" (UID: \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\") " pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" Jul 08 19:12:22.574544 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.574538 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a54fd058-ecef-4fc0-8b95-5b24396b78d1-workload-socket\") pod \"e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n\" (UID: \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\") " pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" Jul 08 19:12:22.574715 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.574564 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a54fd058-ecef-4fc0-8b95-5b24396b78d1-workload-certs\") pod \"e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n\" (UID: \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\") " pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" Jul 08 19:12:22.574715 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.574595 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn8j6\" (UniqueName: \"kubernetes.io/projected/a54fd058-ecef-4fc0-8b95-5b24396b78d1-kube-api-access-dn8j6\") pod \"e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n\" (UID: \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\") " pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" Jul 08 19:12:22.574715 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.574658 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a54fd058-ecef-4fc0-8b95-5b24396b78d1-istio-podinfo\") pod \"e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n\" (UID: \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\") " pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" Jul 08 19:12:22.574715 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.574690 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a54fd058-ecef-4fc0-8b95-5b24396b78d1-istio-data\") pod \"e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n\" (UID: \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\") " pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" Jul 08 19:12:22.595435 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.595408 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n"] Jul 08 19:12:22.675142 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.675107 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a54fd058-ecef-4fc0-8b95-5b24396b78d1-credential-socket\") pod \"e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n\" (UID: \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\") " pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" Jul 08 19:12:22.675142 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.675144 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a54fd058-ecef-4fc0-8b95-5b24396b78d1-istio-envoy\") pod \"e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n\" (UID: \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\") " pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" Jul 08 19:12:22.675434 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.675285 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a54fd058-ecef-4fc0-8b95-5b24396b78d1-workload-socket\") pod \"e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n\" (UID: \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\") " pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" Jul 08 19:12:22.675434 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.675319 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a54fd058-ecef-4fc0-8b95-5b24396b78d1-workload-certs\") pod \"e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n\" (UID: \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\") " pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" Jul 08 19:12:22.675434 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.675348 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn8j6\" (UniqueName: \"kubernetes.io/projected/a54fd058-ecef-4fc0-8b95-5b24396b78d1-kube-api-access-dn8j6\") pod \"e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n\" (UID: \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\") " pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" Jul 08 19:12:22.675434 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.675400 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a54fd058-ecef-4fc0-8b95-5b24396b78d1-istio-podinfo\") pod \"e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n\" (UID: \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\") " pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" Jul 08 19:12:22.675667 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.675440 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a54fd058-ecef-4fc0-8b95-5b24396b78d1-istio-data\") pod \"e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n\" (UID: \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\") " pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" Jul 08 19:12:22.675667 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.675471 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a54fd058-ecef-4fc0-8b95-5b24396b78d1-istio-token\") pod \"e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n\" (UID: \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\") " pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" Jul 08 19:12:22.675667 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.675541 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a54fd058-ecef-4fc0-8b95-5b24396b78d1-credential-socket\") pod \"e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n\" (UID: \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\") " pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" Jul 08 19:12:22.675667 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.675561 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a54fd058-ecef-4fc0-8b95-5b24396b78d1-istiod-ca-cert\") pod \"e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n\" (UID: \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\") " pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" Jul 08 19:12:22.675667 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.675595 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a54fd058-ecef-4fc0-8b95-5b24396b78d1-workload-socket\") pod \"e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n\" (UID: \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\") " pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" Jul 08 19:12:22.675974 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.675667 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a54fd058-ecef-4fc0-8b95-5b24396b78d1-workload-certs\") pod \"e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n\" (UID: \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\") " pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" Jul 08 19:12:22.675974 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.675911 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a54fd058-ecef-4fc0-8b95-5b24396b78d1-istio-data\") pod \"e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n\" (UID: \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\") " pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" Jul 08 19:12:22.676155 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.676137 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a54fd058-ecef-4fc0-8b95-5b24396b78d1-istiod-ca-cert\") pod \"e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n\" (UID: \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\") " pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" Jul 08 19:12:22.677582 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.677559 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a54fd058-ecef-4fc0-8b95-5b24396b78d1-istio-envoy\") pod \"e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n\" (UID: \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\") " pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" Jul 08 19:12:22.677794 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.677775 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a54fd058-ecef-4fc0-8b95-5b24396b78d1-istio-podinfo\") pod \"e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n\" (UID: \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\") " pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" Jul 08 19:12:22.694151 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.694127 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a54fd058-ecef-4fc0-8b95-5b24396b78d1-istio-token\") pod \"e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n\" (UID: \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\") " pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" Jul 08 19:12:22.703446 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.703422 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn8j6\" (UniqueName: \"kubernetes.io/projected/a54fd058-ecef-4fc0-8b95-5b24396b78d1-kube-api-access-dn8j6\") pod \"e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n\" (UID: \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\") " pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" Jul 08 19:12:22.857503 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:22.857417 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" Jul 08 19:12:23.104153 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:23.104124 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n"] Jul 08 19:12:23.105841 ip-10-0-130-48 kubenswrapper[2570]: W0708 19:12:23.105815 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda54fd058_ecef_4fc0_8b95_5b24396b78d1.slice/crio-d925bb4b6b58ac8503007f0649c3472d78874063c60deccc56d750e2138f55d8 WatchSource:0}: Error finding container d925bb4b6b58ac8503007f0649c3472d78874063c60deccc56d750e2138f55d8: Status 404 returned error can't find the container with id d925bb4b6b58ac8503007f0649c3472d78874063c60deccc56d750e2138f55d8 Jul 08 19:12:23.107876 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:23.107823 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:12:23.107921 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:23.107889 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:12:23.107956 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:23.107918 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:12:23.429586 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:23.429545 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:12:23.434454 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:23.434409 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" event={"ID":"a54fd058-ecef-4fc0-8b95-5b24396b78d1","Type":"ContainerStarted","Data":"12230f48885f15d17eab1fc05c1a489d4702cfbcbdfd40405a40d7d4f9675b53"} Jul 08 19:12:23.434600 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:23.434472 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" event={"ID":"a54fd058-ecef-4fc0-8b95-5b24396b78d1","Type":"ContainerStarted","Data":"d925bb4b6b58ac8503007f0649c3472d78874063c60deccc56d750e2138f55d8"} Jul 08 19:12:23.474637 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:23.474581 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" podStartSLOduration=1.4745624689999999 podStartE2EDuration="1.474562469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:12:23.473447603 +0000 UTC m=+1604.320722268" watchObservedRunningTime="2026-07-08 19:12:23.474562469 +0000 UTC m=+1604.321837127" Jul 08 19:12:23.858105 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:23.858025 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" Jul 08 19:12:24.348581 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:24.348549 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs"] Jul 08 19:12:24.351265 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:24.351230 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs" Jul 08 19:12:24.357215 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:24.357197 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s5q9r\"" Jul 08 19:12:24.357792 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:24.357773 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-036c61dd-serving-cert\"" Jul 08 19:12:24.380278 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:24.380232 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs"] Jul 08 19:12:24.392130 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:24.392103 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/044ace3e-65d2-447e-8b7d-b46721dd8c68-openshift-service-ca\") pod \"maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs\" (UID: \"044ace3e-65d2-447e-8b7d-b46721dd8c68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs" Jul 08 19:12:24.392264 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:24.392147 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/044ace3e-65d2-447e-8b7d-b46721dd8c68-maas-api-tls\") pod \"maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs\" (UID: \"044ace3e-65d2-447e-8b7d-b46721dd8c68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs" Jul 08 19:12:24.392264 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:24.392188 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppbdc\" (UniqueName: \"kubernetes.io/projected/044ace3e-65d2-447e-8b7d-b46721dd8c68-kube-api-access-ppbdc\") pod \"maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs\" (UID: \"044ace3e-65d2-447e-8b7d-b46721dd8c68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs" Jul 08 19:12:24.493660 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:24.493626 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/044ace3e-65d2-447e-8b7d-b46721dd8c68-openshift-service-ca\") pod \"maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs\" (UID: \"044ace3e-65d2-447e-8b7d-b46721dd8c68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs" Jul 08 19:12:24.493660 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:24.493660 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/044ace3e-65d2-447e-8b7d-b46721dd8c68-maas-api-tls\") pod \"maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs\" (UID: \"044ace3e-65d2-447e-8b7d-b46721dd8c68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs" Jul 08 19:12:24.493924 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:24.493696 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppbdc\" (UniqueName: \"kubernetes.io/projected/044ace3e-65d2-447e-8b7d-b46721dd8c68-kube-api-access-ppbdc\") pod \"maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs\" (UID: \"044ace3e-65d2-447e-8b7d-b46721dd8c68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs" Jul 08 19:12:24.494033 ip-10-0-130-48 kubenswrapper[2570]: E0708 19:12:24.494006 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-036c61dd-serving-cert: secret "maas-api-e2e-ait-036c61dd-serving-cert" not found Jul 08 19:12:24.494164 ip-10-0-130-48 kubenswrapper[2570]: E0708 19:12:24.494095 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/044ace3e-65d2-447e-8b7d-b46721dd8c68-maas-api-tls podName:044ace3e-65d2-447e-8b7d-b46721dd8c68 nodeName:}" failed. No retries permitted until 2026-07-08 19:12:24.994073647 +0000 UTC m=+1605.841348301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/044ace3e-65d2-447e-8b7d-b46721dd8c68-maas-api-tls") pod "maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs" (UID: "044ace3e-65d2-447e-8b7d-b46721dd8c68") : secret "maas-api-e2e-ait-036c61dd-serving-cert" not found Jul 08 19:12:24.494559 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:24.494538 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/044ace3e-65d2-447e-8b7d-b46721dd8c68-openshift-service-ca\") pod \"maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs\" (UID: \"044ace3e-65d2-447e-8b7d-b46721dd8c68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs" Jul 08 19:12:24.510469 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:24.510449 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppbdc\" (UniqueName: \"kubernetes.io/projected/044ace3e-65d2-447e-8b7d-b46721dd8c68-kube-api-access-ppbdc\") pod \"maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs\" (UID: \"044ace3e-65d2-447e-8b7d-b46721dd8c68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs" Jul 08 19:12:24.863633 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:24.863599 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" Jul 08 19:12:24.998566 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:24.998530 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/044ace3e-65d2-447e-8b7d-b46721dd8c68-maas-api-tls\") pod \"maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs\" (UID: \"044ace3e-65d2-447e-8b7d-b46721dd8c68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs" Jul 08 19:12:25.000675 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:25.000639 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/044ace3e-65d2-447e-8b7d-b46721dd8c68-maas-api-tls\") pod \"maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs\" (UID: \"044ace3e-65d2-447e-8b7d-b46721dd8c68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs" Jul 08 19:12:25.264211 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:25.264115 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs" Jul 08 19:12:25.425808 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:25.425785 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs"] Jul 08 19:12:25.427667 ip-10-0-130-48 kubenswrapper[2570]: W0708 19:12:25.427634 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod044ace3e_65d2_447e_8b7d_b46721dd8c68.slice/crio-9c595b2041d9931cb1bc266628ff03d494ddda11e95be905cbcc48b4a2136c21 WatchSource:0}: Error finding container 9c595b2041d9931cb1bc266628ff03d494ddda11e95be905cbcc48b4a2136c21: Status 404 returned error can't find the container with id 9c595b2041d9931cb1bc266628ff03d494ddda11e95be905cbcc48b4a2136c21 Jul 08 19:12:25.447303 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:25.447274 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs" event={"ID":"044ace3e-65d2-447e-8b7d-b46721dd8c68","Type":"ContainerStarted","Data":"9c595b2041d9931cb1bc266628ff03d494ddda11e95be905cbcc48b4a2136c21"} Jul 08 19:12:25.447494 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:25.447477 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" Jul 08 19:12:25.448431 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:25.448406 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" Jul 08 19:12:26.454710 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:26.454675 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs" event={"ID":"044ace3e-65d2-447e-8b7d-b46721dd8c68","Type":"ContainerStarted","Data":"067081a5bee2556052e27056a42162bdbc2009c23e47b22cd1b00f83c0fbc372"} Jul 08 19:12:26.494038 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:26.493990 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs" podStartSLOduration=2.146352035 podStartE2EDuration="2.493973109s" totalImagesPullingTime="347.621074ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:12:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:12:26.492222774 +0000 UTC m=+1607.339497441" watchObservedRunningTime="2026-07-08 19:12:26.493973109 +0000 UTC m=+1607.341247775" Jul 08 19:12:27.461273 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:27.461222 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs" Jul 08 19:12:29.442778 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:29.442744 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs"] Jul 08 19:12:29.471722 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:29.471680 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs" podUID="044ace3e-65d2-447e-8b7d-b46721dd8c68" containerName="maas-api" containerID="cri-o://067081a5bee2556052e27056a42162bdbc2009c23e47b22cd1b00f83c0fbc372" gracePeriod=30 Jul 08 19:12:29.768350 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:29.768316 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs" Jul 08 19:12:29.853583 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:29.853552 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/044ace3e-65d2-447e-8b7d-b46721dd8c68-openshift-service-ca\") pod \"044ace3e-65d2-447e-8b7d-b46721dd8c68\" (UID: \"044ace3e-65d2-447e-8b7d-b46721dd8c68\") " Jul 08 19:12:29.854195 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:29.854174 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppbdc\" (UniqueName: \"kubernetes.io/projected/044ace3e-65d2-447e-8b7d-b46721dd8c68-kube-api-access-ppbdc\") pod \"044ace3e-65d2-447e-8b7d-b46721dd8c68\" (UID: \"044ace3e-65d2-447e-8b7d-b46721dd8c68\") " Jul 08 19:12:29.854532 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:29.854515 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/044ace3e-65d2-447e-8b7d-b46721dd8c68-maas-api-tls\") pod \"044ace3e-65d2-447e-8b7d-b46721dd8c68\" (UID: \"044ace3e-65d2-447e-8b7d-b46721dd8c68\") " Jul 08 19:12:29.854910 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:29.854114 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/044ace3e-65d2-447e-8b7d-b46721dd8c68-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "044ace3e-65d2-447e-8b7d-b46721dd8c68" (UID: "044ace3e-65d2-447e-8b7d-b46721dd8c68"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:12:29.859778 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:29.859750 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/044ace3e-65d2-447e-8b7d-b46721dd8c68-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "044ace3e-65d2-447e-8b7d-b46721dd8c68" (UID: "044ace3e-65d2-447e-8b7d-b46721dd8c68"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:12:29.860762 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:29.860734 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/044ace3e-65d2-447e-8b7d-b46721dd8c68-kube-api-access-ppbdc" (OuterVolumeSpecName: "kube-api-access-ppbdc") pod "044ace3e-65d2-447e-8b7d-b46721dd8c68" (UID: "044ace3e-65d2-447e-8b7d-b46721dd8c68"). InnerVolumeSpecName "kube-api-access-ppbdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:12:29.956488 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:29.956434 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/044ace3e-65d2-447e-8b7d-b46721dd8c68-maas-api-tls\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:12:29.956488 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:29.956484 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/044ace3e-65d2-447e-8b7d-b46721dd8c68-openshift-service-ca\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:12:29.956766 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:29.956502 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppbdc\" (UniqueName: \"kubernetes.io/projected/044ace3e-65d2-447e-8b7d-b46721dd8c68-kube-api-access-ppbdc\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:12:30.478959 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:30.478882 2570 generic.go:356] "Generic (PLEG): container finished" podID="044ace3e-65d2-447e-8b7d-b46721dd8c68" containerID="067081a5bee2556052e27056a42162bdbc2009c23e47b22cd1b00f83c0fbc372" exitCode=0 Jul 08 19:12:30.478959 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:30.478932 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs" event={"ID":"044ace3e-65d2-447e-8b7d-b46721dd8c68","Type":"ContainerDied","Data":"067081a5bee2556052e27056a42162bdbc2009c23e47b22cd1b00f83c0fbc372"} Jul 08 19:12:30.479463 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:30.478970 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs" event={"ID":"044ace3e-65d2-447e-8b7d-b46721dd8c68","Type":"ContainerDied","Data":"9c595b2041d9931cb1bc266628ff03d494ddda11e95be905cbcc48b4a2136c21"} Jul 08 19:12:30.479463 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:30.478991 2570 scope.go:117] "RemoveContainer" containerID="067081a5bee2556052e27056a42162bdbc2009c23e47b22cd1b00f83c0fbc372" Jul 08 19:12:30.479463 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:30.478932 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs" Jul 08 19:12:30.491018 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:30.491000 2570 scope.go:117] "RemoveContainer" containerID="067081a5bee2556052e27056a42162bdbc2009c23e47b22cd1b00f83c0fbc372" Jul 08 19:12:30.491287 ip-10-0-130-48 kubenswrapper[2570]: E0708 19:12:30.491266 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"067081a5bee2556052e27056a42162bdbc2009c23e47b22cd1b00f83c0fbc372\": container with ID starting with 067081a5bee2556052e27056a42162bdbc2009c23e47b22cd1b00f83c0fbc372 not found: ID does not exist" containerID="067081a5bee2556052e27056a42162bdbc2009c23e47b22cd1b00f83c0fbc372" Jul 08 19:12:30.491336 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:30.491298 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"067081a5bee2556052e27056a42162bdbc2009c23e47b22cd1b00f83c0fbc372"} err="failed to get container status \"067081a5bee2556052e27056a42162bdbc2009c23e47b22cd1b00f83c0fbc372\": rpc error: code = NotFound desc = could not find container \"067081a5bee2556052e27056a42162bdbc2009c23e47b22cd1b00f83c0fbc372\": container with ID starting with 067081a5bee2556052e27056a42162bdbc2009c23e47b22cd1b00f83c0fbc372 not found: ID does not exist" Jul 08 19:12:30.505268 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:30.505227 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs"] Jul 08 19:12:30.511740 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:30.511720 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-036c61dd-64b94b6c9f-b4pzs"] Jul 08 19:12:31.690836 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:31.690805 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="044ace3e-65d2-447e-8b7d-b46721dd8c68" path="/var/lib/kubelet/pods/044ace3e-65d2-447e-8b7d-b46721dd8c68/volumes" Jul 08 19:12:35.060228 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:35.060193 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n"] Jul 08 19:12:35.060664 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:35.060408 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" podUID="a54fd058-ecef-4fc0-8b95-5b24396b78d1" containerName="istio-proxy" containerID="cri-o://12230f48885f15d17eab1fc05c1a489d4702cfbcbdfd40405a40d7d4f9675b53" gracePeriod=30 Jul 08 19:12:36.919851 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:36.919787 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" Jul 08 19:12:37.025556 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.025525 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dn8j6\" (UniqueName: \"kubernetes.io/projected/a54fd058-ecef-4fc0-8b95-5b24396b78d1-kube-api-access-dn8j6\") pod \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\" (UID: \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\") " Jul 08 19:12:37.025756 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.025574 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a54fd058-ecef-4fc0-8b95-5b24396b78d1-istio-envoy\") pod \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\" (UID: \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\") " Jul 08 19:12:37.025756 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.025627 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a54fd058-ecef-4fc0-8b95-5b24396b78d1-istio-podinfo\") pod \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\" (UID: \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\") " Jul 08 19:12:37.025756 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.025656 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a54fd058-ecef-4fc0-8b95-5b24396b78d1-istio-token\") pod \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\" (UID: \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\") " Jul 08 19:12:37.025756 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.025682 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a54fd058-ecef-4fc0-8b95-5b24396b78d1-workload-certs\") pod \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\" (UID: \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\") " Jul 08 19:12:37.025756 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.025734 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a54fd058-ecef-4fc0-8b95-5b24396b78d1-credential-socket\") pod \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\" (UID: \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\") " Jul 08 19:12:37.026017 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.025758 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a54fd058-ecef-4fc0-8b95-5b24396b78d1-istio-data\") pod \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\" (UID: \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\") " Jul 08 19:12:37.026017 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.025784 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a54fd058-ecef-4fc0-8b95-5b24396b78d1-istiod-ca-cert\") pod \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\" (UID: \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\") " Jul 08 19:12:37.026017 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.025859 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a54fd058-ecef-4fc0-8b95-5b24396b78d1-workload-socket\") pod \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\" (UID: \"a54fd058-ecef-4fc0-8b95-5b24396b78d1\") " Jul 08 19:12:37.026164 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.026039 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a54fd058-ecef-4fc0-8b95-5b24396b78d1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a54fd058-ecef-4fc0-8b95-5b24396b78d1" (UID: "a54fd058-ecef-4fc0-8b95-5b24396b78d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:12:37.026292 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.026244 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a54fd058-ecef-4fc0-8b95-5b24396b78d1-workload-certs\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:12:37.026388 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.026300 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a54fd058-ecef-4fc0-8b95-5b24396b78d1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a54fd058-ecef-4fc0-8b95-5b24396b78d1" (UID: "a54fd058-ecef-4fc0-8b95-5b24396b78d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:12:37.026562 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.026533 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a54fd058-ecef-4fc0-8b95-5b24396b78d1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a54fd058-ecef-4fc0-8b95-5b24396b78d1" (UID: "a54fd058-ecef-4fc0-8b95-5b24396b78d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:12:37.026646 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.026593 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a54fd058-ecef-4fc0-8b95-5b24396b78d1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a54fd058-ecef-4fc0-8b95-5b24396b78d1" (UID: "a54fd058-ecef-4fc0-8b95-5b24396b78d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:12:37.026758 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.026733 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a54fd058-ecef-4fc0-8b95-5b24396b78d1-istio-data" (OuterVolumeSpecName: "istio-data") pod "a54fd058-ecef-4fc0-8b95-5b24396b78d1" (UID: "a54fd058-ecef-4fc0-8b95-5b24396b78d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:12:37.027817 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.027789 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a54fd058-ecef-4fc0-8b95-5b24396b78d1-kube-api-access-dn8j6" (OuterVolumeSpecName: "kube-api-access-dn8j6") pod "a54fd058-ecef-4fc0-8b95-5b24396b78d1" (UID: "a54fd058-ecef-4fc0-8b95-5b24396b78d1"). InnerVolumeSpecName "kube-api-access-dn8j6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:12:37.027922 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.027862 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a54fd058-ecef-4fc0-8b95-5b24396b78d1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a54fd058-ecef-4fc0-8b95-5b24396b78d1" (UID: "a54fd058-ecef-4fc0-8b95-5b24396b78d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 19:12:37.028112 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.028086 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a54fd058-ecef-4fc0-8b95-5b24396b78d1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a54fd058-ecef-4fc0-8b95-5b24396b78d1" (UID: "a54fd058-ecef-4fc0-8b95-5b24396b78d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:12:37.028112 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.028097 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a54fd058-ecef-4fc0-8b95-5b24396b78d1-istio-token" (OuterVolumeSpecName: "istio-token") pod "a54fd058-ecef-4fc0-8b95-5b24396b78d1" (UID: "a54fd058-ecef-4fc0-8b95-5b24396b78d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:12:37.127337 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.127304 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a54fd058-ecef-4fc0-8b95-5b24396b78d1-credential-socket\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:12:37.127337 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.127330 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a54fd058-ecef-4fc0-8b95-5b24396b78d1-istio-data\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:12:37.127337 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.127340 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a54fd058-ecef-4fc0-8b95-5b24396b78d1-istiod-ca-cert\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:12:37.127337 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.127348 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a54fd058-ecef-4fc0-8b95-5b24396b78d1-workload-socket\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:12:37.127587 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.127356 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn8j6\" (UniqueName: \"kubernetes.io/projected/a54fd058-ecef-4fc0-8b95-5b24396b78d1-kube-api-access-dn8j6\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:12:37.127587 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.127364 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a54fd058-ecef-4fc0-8b95-5b24396b78d1-istio-envoy\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:12:37.127587 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.127372 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a54fd058-ecef-4fc0-8b95-5b24396b78d1-istio-podinfo\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:12:37.127587 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.127380 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a54fd058-ecef-4fc0-8b95-5b24396b78d1-istio-token\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:12:37.522308 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.522275 2570 generic.go:356] "Generic (PLEG): container finished" podID="a54fd058-ecef-4fc0-8b95-5b24396b78d1" containerID="12230f48885f15d17eab1fc05c1a489d4702cfbcbdfd40405a40d7d4f9675b53" exitCode=0 Jul 08 19:12:37.522493 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.522324 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" Jul 08 19:12:37.522493 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.522362 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" event={"ID":"a54fd058-ecef-4fc0-8b95-5b24396b78d1","Type":"ContainerDied","Data":"12230f48885f15d17eab1fc05c1a489d4702cfbcbdfd40405a40d7d4f9675b53"} Jul 08 19:12:37.522493 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.522421 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n" event={"ID":"a54fd058-ecef-4fc0-8b95-5b24396b78d1","Type":"ContainerDied","Data":"d925bb4b6b58ac8503007f0649c3472d78874063c60deccc56d750e2138f55d8"} Jul 08 19:12:37.522493 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.522446 2570 scope.go:117] "RemoveContainer" containerID="12230f48885f15d17eab1fc05c1a489d4702cfbcbdfd40405a40d7d4f9675b53" Jul 08 19:12:37.533345 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.533310 2570 scope.go:117] "RemoveContainer" containerID="12230f48885f15d17eab1fc05c1a489d4702cfbcbdfd40405a40d7d4f9675b53" Jul 08 19:12:37.533700 ip-10-0-130-48 kubenswrapper[2570]: E0708 19:12:37.533675 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12230f48885f15d17eab1fc05c1a489d4702cfbcbdfd40405a40d7d4f9675b53\": container with ID starting with 12230f48885f15d17eab1fc05c1a489d4702cfbcbdfd40405a40d7d4f9675b53 not found: ID does not exist" containerID="12230f48885f15d17eab1fc05c1a489d4702cfbcbdfd40405a40d7d4f9675b53" Jul 08 19:12:37.533838 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.533707 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12230f48885f15d17eab1fc05c1a489d4702cfbcbdfd40405a40d7d4f9675b53"} err="failed to get container status \"12230f48885f15d17eab1fc05c1a489d4702cfbcbdfd40405a40d7d4f9675b53\": rpc error: code = NotFound desc = could not find container \"12230f48885f15d17eab1fc05c1a489d4702cfbcbdfd40405a40d7d4f9675b53\": container with ID starting with 12230f48885f15d17eab1fc05c1a489d4702cfbcbdfd40405a40d7d4f9675b53 not found: ID does not exist" Jul 08 19:12:37.551066 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.551037 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n"] Jul 08 19:12:37.554811 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.554784 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-036c61dd-openshift-default-6b5f7c8ff7-xh55n"] Jul 08 19:12:37.690773 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:37.690735 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a54fd058-ecef-4fc0-8b95-5b24396b78d1" path="/var/lib/kubelet/pods/a54fd058-ecef-4fc0-8b95-5b24396b78d1/volumes" Jul 08 19:12:41.887235 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:41.887205 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb"] Jul 08 19:12:41.887637 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:41.887622 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="044ace3e-65d2-447e-8b7d-b46721dd8c68" containerName="maas-api" Jul 08 19:12:41.887696 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:41.887639 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="044ace3e-65d2-447e-8b7d-b46721dd8c68" containerName="maas-api" Jul 08 19:12:41.887696 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:41.887662 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a54fd058-ecef-4fc0-8b95-5b24396b78d1" containerName="istio-proxy" Jul 08 19:12:41.887696 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:41.887669 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a54fd058-ecef-4fc0-8b95-5b24396b78d1" containerName="istio-proxy" Jul 08 19:12:41.887790 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:41.887734 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="044ace3e-65d2-447e-8b7d-b46721dd8c68" containerName="maas-api" Jul 08 19:12:41.887790 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:41.887745 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a54fd058-ecef-4fc0-8b95-5b24396b78d1" containerName="istio-proxy" Jul 08 19:12:41.891914 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:41.891895 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" Jul 08 19:12:41.897470 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:41.897449 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-6bcda1d3-openshift-default-dockercfg-wtr5q\"" Jul 08 19:12:41.913508 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:41.913483 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb"] Jul 08 19:12:41.969333 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:41.969301 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mczhs\" (UniqueName: \"kubernetes.io/projected/c8f22cda-f778-43a2-8507-24cb0277a83a-kube-api-access-mczhs\") pod \"e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb\" (UID: \"c8f22cda-f778-43a2-8507-24cb0277a83a\") " pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" Jul 08 19:12:41.969485 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:41.969347 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f22cda-f778-43a2-8507-24cb0277a83a-credential-socket\") pod \"e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb\" (UID: \"c8f22cda-f778-43a2-8507-24cb0277a83a\") " pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" Jul 08 19:12:41.969485 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:41.969412 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8f22cda-f778-43a2-8507-24cb0277a83a-workload-certs\") pod \"e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb\" (UID: \"c8f22cda-f778-43a2-8507-24cb0277a83a\") " pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" Jul 08 19:12:41.969485 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:41.969439 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f22cda-f778-43a2-8507-24cb0277a83a-workload-socket\") pod \"e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb\" (UID: \"c8f22cda-f778-43a2-8507-24cb0277a83a\") " pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" Jul 08 19:12:41.969485 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:41.969463 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8f22cda-f778-43a2-8507-24cb0277a83a-istio-envoy\") pod \"e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb\" (UID: \"c8f22cda-f778-43a2-8507-24cb0277a83a\") " pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" Jul 08 19:12:41.969485 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:41.969481 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8f22cda-f778-43a2-8507-24cb0277a83a-istio-token\") pod \"e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb\" (UID: \"c8f22cda-f778-43a2-8507-24cb0277a83a\") " pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" Jul 08 19:12:41.969692 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:41.969510 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8f22cda-f778-43a2-8507-24cb0277a83a-istio-data\") pod \"e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb\" (UID: \"c8f22cda-f778-43a2-8507-24cb0277a83a\") " pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" Jul 08 19:12:41.969692 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:41.969549 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8f22cda-f778-43a2-8507-24cb0277a83a-istio-podinfo\") pod \"e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb\" (UID: \"c8f22cda-f778-43a2-8507-24cb0277a83a\") " pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" Jul 08 19:12:41.969692 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:41.969586 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8f22cda-f778-43a2-8507-24cb0277a83a-istiod-ca-cert\") pod \"e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb\" (UID: \"c8f22cda-f778-43a2-8507-24cb0277a83a\") " pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" Jul 08 19:12:42.070509 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:42.070476 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8f22cda-f778-43a2-8507-24cb0277a83a-istio-podinfo\") pod \"e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb\" (UID: \"c8f22cda-f778-43a2-8507-24cb0277a83a\") " pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" Jul 08 19:12:42.070509 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:42.070511 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8f22cda-f778-43a2-8507-24cb0277a83a-istiod-ca-cert\") pod \"e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb\" (UID: \"c8f22cda-f778-43a2-8507-24cb0277a83a\") " pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" Jul 08 19:12:42.070729 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:42.070583 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mczhs\" (UniqueName: \"kubernetes.io/projected/c8f22cda-f778-43a2-8507-24cb0277a83a-kube-api-access-mczhs\") pod \"e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb\" (UID: \"c8f22cda-f778-43a2-8507-24cb0277a83a\") " pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" Jul 08 19:12:42.070729 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:42.070605 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f22cda-f778-43a2-8507-24cb0277a83a-credential-socket\") pod \"e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb\" (UID: \"c8f22cda-f778-43a2-8507-24cb0277a83a\") " pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" Jul 08 19:12:42.070729 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:42.070632 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8f22cda-f778-43a2-8507-24cb0277a83a-workload-certs\") pod \"e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb\" (UID: \"c8f22cda-f778-43a2-8507-24cb0277a83a\") " pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" Jul 08 19:12:42.070881 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:42.070746 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f22cda-f778-43a2-8507-24cb0277a83a-workload-socket\") pod \"e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb\" (UID: \"c8f22cda-f778-43a2-8507-24cb0277a83a\") " pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" Jul 08 19:12:42.070881 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:42.070780 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8f22cda-f778-43a2-8507-24cb0277a83a-istio-envoy\") pod \"e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb\" (UID: \"c8f22cda-f778-43a2-8507-24cb0277a83a\") " pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" Jul 08 19:12:42.070881 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:42.070804 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8f22cda-f778-43a2-8507-24cb0277a83a-istio-token\") pod \"e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb\" (UID: \"c8f22cda-f778-43a2-8507-24cb0277a83a\") " pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" Jul 08 19:12:42.070881 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:42.070860 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8f22cda-f778-43a2-8507-24cb0277a83a-istio-data\") pod \"e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb\" (UID: \"c8f22cda-f778-43a2-8507-24cb0277a83a\") " pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" Jul 08 19:12:42.071079 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:42.071060 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f22cda-f778-43a2-8507-24cb0277a83a-credential-socket\") pod \"e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb\" (UID: \"c8f22cda-f778-43a2-8507-24cb0277a83a\") " pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" Jul 08 19:12:42.071136 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:42.071114 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8f22cda-f778-43a2-8507-24cb0277a83a-workload-certs\") pod \"e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb\" (UID: \"c8f22cda-f778-43a2-8507-24cb0277a83a\") " pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" Jul 08 19:12:42.071216 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:42.071191 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f22cda-f778-43a2-8507-24cb0277a83a-workload-socket\") pod \"e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb\" (UID: \"c8f22cda-f778-43a2-8507-24cb0277a83a\") " pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" Jul 08 19:12:42.071344 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:42.071326 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8f22cda-f778-43a2-8507-24cb0277a83a-istio-data\") pod \"e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb\" (UID: \"c8f22cda-f778-43a2-8507-24cb0277a83a\") " pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" Jul 08 19:12:42.071520 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:42.071501 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8f22cda-f778-43a2-8507-24cb0277a83a-istiod-ca-cert\") pod \"e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb\" (UID: \"c8f22cda-f778-43a2-8507-24cb0277a83a\") " pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" Jul 08 19:12:42.072922 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:42.072891 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8f22cda-f778-43a2-8507-24cb0277a83a-istio-envoy\") pod \"e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb\" (UID: \"c8f22cda-f778-43a2-8507-24cb0277a83a\") " pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" Jul 08 19:12:42.073128 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:42.073109 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8f22cda-f778-43a2-8507-24cb0277a83a-istio-podinfo\") pod \"e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb\" (UID: \"c8f22cda-f778-43a2-8507-24cb0277a83a\") " pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" Jul 08 19:12:42.083578 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:42.083557 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mczhs\" (UniqueName: \"kubernetes.io/projected/c8f22cda-f778-43a2-8507-24cb0277a83a-kube-api-access-mczhs\") pod \"e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb\" (UID: \"c8f22cda-f778-43a2-8507-24cb0277a83a\") " pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" Jul 08 19:12:42.086543 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:42.086523 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8f22cda-f778-43a2-8507-24cb0277a83a-istio-token\") pod \"e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb\" (UID: \"c8f22cda-f778-43a2-8507-24cb0277a83a\") " pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" Jul 08 19:12:42.202559 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:42.202533 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" Jul 08 19:12:42.345688 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:42.345469 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb"] Jul 08 19:12:42.347951 ip-10-0-130-48 kubenswrapper[2570]: W0708 19:12:42.347921 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8f22cda_f778_43a2_8507_24cb0277a83a.slice/crio-7153ebee3589a91789d5d877d162adf2a13a4b81f80d970647185c2d8a173bd3 WatchSource:0}: Error finding container 7153ebee3589a91789d5d877d162adf2a13a4b81f80d970647185c2d8a173bd3: Status 404 returned error can't find the container with id 7153ebee3589a91789d5d877d162adf2a13a4b81f80d970647185c2d8a173bd3 Jul 08 19:12:42.350051 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:42.350017 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:12:42.350119 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:42.350085 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:12:42.350119 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:42.350114 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:12:42.557055 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:42.556953 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" event={"ID":"c8f22cda-f778-43a2-8507-24cb0277a83a","Type":"ContainerStarted","Data":"4c2f93e66a3b4c3ebd9bddce6911c18112e78982d3d1ba851ceb83dc4171dfee"} Jul 08 19:12:42.557055 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:42.557019 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" event={"ID":"c8f22cda-f778-43a2-8507-24cb0277a83a","Type":"ContainerStarted","Data":"7153ebee3589a91789d5d877d162adf2a13a4b81f80d970647185c2d8a173bd3"} Jul 08 19:12:42.584842 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:42.584787 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" podStartSLOduration=1.584767642 podStartE2EDuration="1.584767642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:12:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:12:42.580686657 +0000 UTC m=+1623.427961325" watchObservedRunningTime="2026-07-08 19:12:42.584767642 +0000 UTC m=+1623.432042309" Jul 08 19:12:43.203157 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:43.203117 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" Jul 08 19:12:43.208476 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:43.208442 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" Jul 08 19:12:43.228802 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:43.228765 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6bcda1d3-588cd875b9-nbcqn"] Jul 08 19:12:43.233059 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:43.233037 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6bcda1d3-588cd875b9-nbcqn" Jul 08 19:12:43.236028 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:43.235999 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s5q9r\"" Jul 08 19:12:43.236178 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:43.236111 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-6bcda1d3-serving-cert\"" Jul 08 19:12:43.250475 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:43.250455 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6bcda1d3-588cd875b9-nbcqn"] Jul 08 19:12:43.382869 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:43.382837 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk84h\" (UniqueName: \"kubernetes.io/projected/cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9-kube-api-access-sk84h\") pod \"maas-api-e2e-derive-6bcda1d3-588cd875b9-nbcqn\" (UID: \"cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6bcda1d3-588cd875b9-nbcqn" Jul 08 19:12:43.382869 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:43.382871 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9-maas-api-tls\") pod \"maas-api-e2e-derive-6bcda1d3-588cd875b9-nbcqn\" (UID: \"cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6bcda1d3-588cd875b9-nbcqn" Jul 08 19:12:43.383061 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:43.382895 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9-openshift-service-ca\") pod \"maas-api-e2e-derive-6bcda1d3-588cd875b9-nbcqn\" (UID: \"cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6bcda1d3-588cd875b9-nbcqn" Jul 08 19:12:43.483699 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:43.483620 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk84h\" (UniqueName: \"kubernetes.io/projected/cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9-kube-api-access-sk84h\") pod \"maas-api-e2e-derive-6bcda1d3-588cd875b9-nbcqn\" (UID: \"cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6bcda1d3-588cd875b9-nbcqn" Jul 08 19:12:43.483699 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:43.483656 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9-maas-api-tls\") pod \"maas-api-e2e-derive-6bcda1d3-588cd875b9-nbcqn\" (UID: \"cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6bcda1d3-588cd875b9-nbcqn" Jul 08 19:12:43.483699 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:43.483680 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9-openshift-service-ca\") pod \"maas-api-e2e-derive-6bcda1d3-588cd875b9-nbcqn\" (UID: \"cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6bcda1d3-588cd875b9-nbcqn" Jul 08 19:12:43.484385 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:43.484361 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9-openshift-service-ca\") pod \"maas-api-e2e-derive-6bcda1d3-588cd875b9-nbcqn\" (UID: \"cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6bcda1d3-588cd875b9-nbcqn" Jul 08 19:12:43.486108 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:43.486082 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9-maas-api-tls\") pod \"maas-api-e2e-derive-6bcda1d3-588cd875b9-nbcqn\" (UID: \"cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6bcda1d3-588cd875b9-nbcqn" Jul 08 19:12:43.497229 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:43.497206 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk84h\" (UniqueName: \"kubernetes.io/projected/cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9-kube-api-access-sk84h\") pod \"maas-api-e2e-derive-6bcda1d3-588cd875b9-nbcqn\" (UID: \"cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6bcda1d3-588cd875b9-nbcqn" Jul 08 19:12:43.549289 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:43.549242 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6bcda1d3-588cd875b9-nbcqn" Jul 08 19:12:43.562653 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:43.562630 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" Jul 08 19:12:43.563863 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:43.563839 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" Jul 08 19:12:43.725622 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:43.725586 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6bcda1d3-588cd875b9-nbcqn"] Jul 08 19:12:43.733109 ip-10-0-130-48 kubenswrapper[2570]: W0708 19:12:43.733081 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdd2e7bf_ad14_404f_b198_fb8cd2c8bdc9.slice/crio-e4fee592dc72a5f1f441554b4088389e12ed1d8cae376f6dfb175af71b70276d WatchSource:0}: Error finding container e4fee592dc72a5f1f441554b4088389e12ed1d8cae376f6dfb175af71b70276d: Status 404 returned error can't find the container with id e4fee592dc72a5f1f441554b4088389e12ed1d8cae376f6dfb175af71b70276d Jul 08 19:12:44.570308 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:44.570193 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6bcda1d3-588cd875b9-nbcqn" event={"ID":"cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9","Type":"ContainerStarted","Data":"a7f462d5e2848eb03d2053659daf23fdd148fbe42c6c757a7f4d4a530418105d"} Jul 08 19:12:44.570308 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:44.570280 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6bcda1d3-588cd875b9-nbcqn" event={"ID":"cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9","Type":"ContainerStarted","Data":"e4fee592dc72a5f1f441554b4088389e12ed1d8cae376f6dfb175af71b70276d"} Jul 08 19:12:44.591091 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:44.591042 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6bcda1d3-588cd875b9-nbcqn" podStartSLOduration=1.203823993 podStartE2EDuration="1.591026568s" totalImagesPullingTime="387.202575ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:12:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:12:44.590481136 +0000 UTC m=+1625.437755802" watchObservedRunningTime="2026-07-08 19:12:44.591026568 +0000 UTC m=+1625.438301275" Jul 08 19:12:45.576752 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:45.576719 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6bcda1d3-588cd875b9-nbcqn" Jul 08 19:12:46.305717 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:46.305687 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:12:47.691128 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:47.691097 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb"] Jul 08 19:12:47.691599 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:47.691491 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" podUID="c8f22cda-f778-43a2-8507-24cb0277a83a" containerName="istio-proxy" containerID="cri-o://4c2f93e66a3b4c3ebd9bddce6911c18112e78982d3d1ba851ceb83dc4171dfee" gracePeriod=30 Jul 08 19:12:48.052331 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:48.043700 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6bcda1d3-588cd875b9-nbcqn"] Jul 08 19:12:48.061326 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:48.058758 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6bcda1d3-588cd875b9-nbcqn" podUID="cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9" containerName="maas-api" containerID="cri-o://a7f462d5e2848eb03d2053659daf23fdd148fbe42c6c757a7f4d4a530418105d" gracePeriod=30 Jul 08 19:12:48.361616 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:48.361590 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6bcda1d3-588cd875b9-nbcqn" Jul 08 19:12:48.428442 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:48.428415 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sk84h\" (UniqueName: \"kubernetes.io/projected/cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9-kube-api-access-sk84h\") pod \"cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9\" (UID: \"cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9\") " Jul 08 19:12:48.428609 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:48.428467 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9-maas-api-tls\") pod \"cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9\" (UID: \"cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9\") " Jul 08 19:12:48.428609 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:48.428519 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9-openshift-service-ca\") pod \"cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9\" (UID: \"cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9\") " Jul 08 19:12:48.428870 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:48.428844 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9" (UID: "cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:12:48.430866 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:48.430839 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9-kube-api-access-sk84h" (OuterVolumeSpecName: "kube-api-access-sk84h") pod "cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9" (UID: "cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9"). InnerVolumeSpecName "kube-api-access-sk84h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:12:48.460801 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:48.460751 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9" (UID: "cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:12:48.529702 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:48.529661 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk84h\" (UniqueName: \"kubernetes.io/projected/cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9-kube-api-access-sk84h\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:12:48.529702 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:48.529706 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9-maas-api-tls\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:12:48.529942 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:48.529721 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9-openshift-service-ca\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:12:48.596389 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:48.596301 2570 generic.go:356] "Generic (PLEG): container finished" podID="cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9" containerID="a7f462d5e2848eb03d2053659daf23fdd148fbe42c6c757a7f4d4a530418105d" exitCode=0 Jul 08 19:12:48.596389 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:48.596359 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6bcda1d3-588cd875b9-nbcqn" Jul 08 19:12:48.596612 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:48.596394 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6bcda1d3-588cd875b9-nbcqn" event={"ID":"cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9","Type":"ContainerDied","Data":"a7f462d5e2848eb03d2053659daf23fdd148fbe42c6c757a7f4d4a530418105d"} Jul 08 19:12:48.596612 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:48.596454 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6bcda1d3-588cd875b9-nbcqn" event={"ID":"cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9","Type":"ContainerDied","Data":"e4fee592dc72a5f1f441554b4088389e12ed1d8cae376f6dfb175af71b70276d"} Jul 08 19:12:48.596612 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:48.596478 2570 scope.go:117] "RemoveContainer" containerID="a7f462d5e2848eb03d2053659daf23fdd148fbe42c6c757a7f4d4a530418105d" Jul 08 19:12:48.605955 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:48.605937 2570 scope.go:117] "RemoveContainer" containerID="a7f462d5e2848eb03d2053659daf23fdd148fbe42c6c757a7f4d4a530418105d" Jul 08 19:12:48.606219 ip-10-0-130-48 kubenswrapper[2570]: E0708 19:12:48.606205 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7f462d5e2848eb03d2053659daf23fdd148fbe42c6c757a7f4d4a530418105d\": container with ID starting with a7f462d5e2848eb03d2053659daf23fdd148fbe42c6c757a7f4d4a530418105d not found: ID does not exist" containerID="a7f462d5e2848eb03d2053659daf23fdd148fbe42c6c757a7f4d4a530418105d" Jul 08 19:12:48.606290 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:48.606226 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7f462d5e2848eb03d2053659daf23fdd148fbe42c6c757a7f4d4a530418105d"} err="failed to get container status \"a7f462d5e2848eb03d2053659daf23fdd148fbe42c6c757a7f4d4a530418105d\": rpc error: code = NotFound desc = could not find container \"a7f462d5e2848eb03d2053659daf23fdd148fbe42c6c757a7f4d4a530418105d\": container with ID starting with a7f462d5e2848eb03d2053659daf23fdd148fbe42c6c757a7f4d4a530418105d not found: ID does not exist" Jul 08 19:12:48.617916 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:48.617896 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6bcda1d3-588cd875b9-nbcqn"] Jul 08 19:12:48.622935 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:48.622915 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6bcda1d3-588cd875b9-nbcqn"] Jul 08 19:12:49.691269 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:49.691224 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9" path="/var/lib/kubelet/pods/cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9/volumes" Jul 08 19:12:54.155090 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.155068 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" Jul 08 19:12:54.291035 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.290956 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8f22cda-f778-43a2-8507-24cb0277a83a-istio-podinfo\") pod \"c8f22cda-f778-43a2-8507-24cb0277a83a\" (UID: \"c8f22cda-f778-43a2-8507-24cb0277a83a\") " Jul 08 19:12:54.291035 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.291018 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f22cda-f778-43a2-8507-24cb0277a83a-credential-socket\") pod \"c8f22cda-f778-43a2-8507-24cb0277a83a\" (UID: \"c8f22cda-f778-43a2-8507-24cb0277a83a\") " Jul 08 19:12:54.291274 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.291058 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f22cda-f778-43a2-8507-24cb0277a83a-workload-socket\") pod \"c8f22cda-f778-43a2-8507-24cb0277a83a\" (UID: \"c8f22cda-f778-43a2-8507-24cb0277a83a\") " Jul 08 19:12:54.291274 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.291076 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8f22cda-f778-43a2-8507-24cb0277a83a-istio-data\") pod \"c8f22cda-f778-43a2-8507-24cb0277a83a\" (UID: \"c8f22cda-f778-43a2-8507-24cb0277a83a\") " Jul 08 19:12:54.291274 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.291099 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8f22cda-f778-43a2-8507-24cb0277a83a-istio-envoy\") pod \"c8f22cda-f778-43a2-8507-24cb0277a83a\" (UID: \"c8f22cda-f778-43a2-8507-24cb0277a83a\") " Jul 08 19:12:54.291274 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.291151 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mczhs\" (UniqueName: \"kubernetes.io/projected/c8f22cda-f778-43a2-8507-24cb0277a83a-kube-api-access-mczhs\") pod \"c8f22cda-f778-43a2-8507-24cb0277a83a\" (UID: \"c8f22cda-f778-43a2-8507-24cb0277a83a\") " Jul 08 19:12:54.291274 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.291177 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8f22cda-f778-43a2-8507-24cb0277a83a-workload-certs\") pod \"c8f22cda-f778-43a2-8507-24cb0277a83a\" (UID: \"c8f22cda-f778-43a2-8507-24cb0277a83a\") " Jul 08 19:12:54.291538 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.291286 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8f22cda-f778-43a2-8507-24cb0277a83a-istio-token\") pod \"c8f22cda-f778-43a2-8507-24cb0277a83a\" (UID: \"c8f22cda-f778-43a2-8507-24cb0277a83a\") " Jul 08 19:12:54.291538 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.291310 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8f22cda-f778-43a2-8507-24cb0277a83a-istiod-ca-cert\") pod \"c8f22cda-f778-43a2-8507-24cb0277a83a\" (UID: \"c8f22cda-f778-43a2-8507-24cb0277a83a\") " Jul 08 19:12:54.291538 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.291449 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8f22cda-f778-43a2-8507-24cb0277a83a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c8f22cda-f778-43a2-8507-24cb0277a83a" (UID: "c8f22cda-f778-43a2-8507-24cb0277a83a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:12:54.291538 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.291489 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8f22cda-f778-43a2-8507-24cb0277a83a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c8f22cda-f778-43a2-8507-24cb0277a83a" (UID: "c8f22cda-f778-43a2-8507-24cb0277a83a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:12:54.291767 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.291627 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8f22cda-f778-43a2-8507-24cb0277a83a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c8f22cda-f778-43a2-8507-24cb0277a83a" (UID: "c8f22cda-f778-43a2-8507-24cb0277a83a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:12:54.291767 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.291756 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8f22cda-f778-43a2-8507-24cb0277a83a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c8f22cda-f778-43a2-8507-24cb0277a83a" (UID: "c8f22cda-f778-43a2-8507-24cb0277a83a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:12:54.291884 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.291764 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f22cda-f778-43a2-8507-24cb0277a83a-credential-socket\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:12:54.291884 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.291788 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f22cda-f778-43a2-8507-24cb0277a83a-workload-socket\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:12:54.291884 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.291803 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8f22cda-f778-43a2-8507-24cb0277a83a-workload-certs\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:12:54.292512 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.292483 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8f22cda-f778-43a2-8507-24cb0277a83a-istio-data" (OuterVolumeSpecName: "istio-data") pod "c8f22cda-f778-43a2-8507-24cb0277a83a" (UID: "c8f22cda-f778-43a2-8507-24cb0277a83a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:12:54.293337 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.293308 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8f22cda-f778-43a2-8507-24cb0277a83a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c8f22cda-f778-43a2-8507-24cb0277a83a" (UID: "c8f22cda-f778-43a2-8507-24cb0277a83a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 19:12:54.293440 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.293417 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8f22cda-f778-43a2-8507-24cb0277a83a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c8f22cda-f778-43a2-8507-24cb0277a83a" (UID: "c8f22cda-f778-43a2-8507-24cb0277a83a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:12:54.293554 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.293537 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8f22cda-f778-43a2-8507-24cb0277a83a-kube-api-access-mczhs" (OuterVolumeSpecName: "kube-api-access-mczhs") pod "c8f22cda-f778-43a2-8507-24cb0277a83a" (UID: "c8f22cda-f778-43a2-8507-24cb0277a83a"). InnerVolumeSpecName "kube-api-access-mczhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:12:54.293554 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.293543 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8f22cda-f778-43a2-8507-24cb0277a83a-istio-token" (OuterVolumeSpecName: "istio-token") pod "c8f22cda-f778-43a2-8507-24cb0277a83a" (UID: "c8f22cda-f778-43a2-8507-24cb0277a83a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:12:54.393223 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.393198 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mczhs\" (UniqueName: \"kubernetes.io/projected/c8f22cda-f778-43a2-8507-24cb0277a83a-kube-api-access-mczhs\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:12:54.393223 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.393221 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8f22cda-f778-43a2-8507-24cb0277a83a-istio-token\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:12:54.393386 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.393230 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8f22cda-f778-43a2-8507-24cb0277a83a-istiod-ca-cert\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:12:54.393386 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.393238 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8f22cda-f778-43a2-8507-24cb0277a83a-istio-podinfo\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:12:54.393386 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.393246 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8f22cda-f778-43a2-8507-24cb0277a83a-istio-data\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:12:54.393386 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.393276 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8f22cda-f778-43a2-8507-24cb0277a83a-istio-envoy\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:12:54.641098 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.641067 2570 generic.go:356] "Generic (PLEG): container finished" podID="c8f22cda-f778-43a2-8507-24cb0277a83a" containerID="4c2f93e66a3b4c3ebd9bddce6911c18112e78982d3d1ba851ceb83dc4171dfee" exitCode=0 Jul 08 19:12:54.641294 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.641140 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" Jul 08 19:12:54.641294 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.641145 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" event={"ID":"c8f22cda-f778-43a2-8507-24cb0277a83a","Type":"ContainerDied","Data":"4c2f93e66a3b4c3ebd9bddce6911c18112e78982d3d1ba851ceb83dc4171dfee"} Jul 08 19:12:54.641428 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.641301 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb" event={"ID":"c8f22cda-f778-43a2-8507-24cb0277a83a","Type":"ContainerDied","Data":"7153ebee3589a91789d5d877d162adf2a13a4b81f80d970647185c2d8a173bd3"} Jul 08 19:12:54.641428 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.641326 2570 scope.go:117] "RemoveContainer" containerID="4c2f93e66a3b4c3ebd9bddce6911c18112e78982d3d1ba851ceb83dc4171dfee" Jul 08 19:12:54.650873 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.650740 2570 scope.go:117] "RemoveContainer" containerID="4c2f93e66a3b4c3ebd9bddce6911c18112e78982d3d1ba851ceb83dc4171dfee" Jul 08 19:12:54.651028 ip-10-0-130-48 kubenswrapper[2570]: E0708 19:12:54.651006 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c2f93e66a3b4c3ebd9bddce6911c18112e78982d3d1ba851ceb83dc4171dfee\": container with ID starting with 4c2f93e66a3b4c3ebd9bddce6911c18112e78982d3d1ba851ceb83dc4171dfee not found: ID does not exist" containerID="4c2f93e66a3b4c3ebd9bddce6911c18112e78982d3d1ba851ceb83dc4171dfee" Jul 08 19:12:54.651086 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.651040 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c2f93e66a3b4c3ebd9bddce6911c18112e78982d3d1ba851ceb83dc4171dfee"} err="failed to get container status \"4c2f93e66a3b4c3ebd9bddce6911c18112e78982d3d1ba851ceb83dc4171dfee\": rpc error: code = NotFound desc = could not find container \"4c2f93e66a3b4c3ebd9bddce6911c18112e78982d3d1ba851ceb83dc4171dfee\": container with ID starting with 4c2f93e66a3b4c3ebd9bddce6911c18112e78982d3d1ba851ceb83dc4171dfee not found: ID does not exist" Jul 08 19:12:54.665786 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.665760 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb"] Jul 08 19:12:54.672555 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:54.672535 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-6bcda1d3-openshift-default-6bb48fc67-sgvjb"] Jul 08 19:12:55.689912 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:55.689880 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8f22cda-f778-43a2-8507-24cb0277a83a" path="/var/lib/kubelet/pods/c8f22cda-f778-43a2-8507-24cb0277a83a/volumes" Jul 08 19:12:58.859596 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:12:58.859559 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:14:20.694848 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:20.694769 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:14:29.494807 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:29.494767 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:14:34.365059 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.365023 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h"] Jul 08 19:14:34.365478 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.365439 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8f22cda-f778-43a2-8507-24cb0277a83a" containerName="istio-proxy" Jul 08 19:14:34.365478 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.365451 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8f22cda-f778-43a2-8507-24cb0277a83a" containerName="istio-proxy" Jul 08 19:14:34.365478 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.365471 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9" containerName="maas-api" Jul 08 19:14:34.365478 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.365476 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9" containerName="maas-api" Jul 08 19:14:34.365611 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.365542 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8f22cda-f778-43a2-8507-24cb0277a83a" containerName="istio-proxy" Jul 08 19:14:34.365611 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.365551 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdd2e7bf-ad14-404f-b198-fb8cd2c8bdc9" containerName="maas-api" Jul 08 19:14:34.368935 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.368914 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:34.371611 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.371588 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-164a1c52-openshift-default-dockercfg-k4ncv\"" Jul 08 19:14:34.396311 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.396284 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h"] Jul 08 19:14:34.487989 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.487946 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c25889c-7caa-4718-b251-5231794a7ccd-istio-envoy\") pod \"e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h\" (UID: \"2c25889c-7caa-4718-b251-5231794a7ccd\") " pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:34.487989 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.487998 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gtng\" (UniqueName: \"kubernetes.io/projected/2c25889c-7caa-4718-b251-5231794a7ccd-kube-api-access-4gtng\") pod \"e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h\" (UID: \"2c25889c-7caa-4718-b251-5231794a7ccd\") " pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:34.488248 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.488074 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c25889c-7caa-4718-b251-5231794a7ccd-istio-data\") pod \"e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h\" (UID: \"2c25889c-7caa-4718-b251-5231794a7ccd\") " pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:34.488248 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.488106 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c25889c-7caa-4718-b251-5231794a7ccd-workload-certs\") pod \"e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h\" (UID: \"2c25889c-7caa-4718-b251-5231794a7ccd\") " pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:34.488248 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.488132 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c25889c-7caa-4718-b251-5231794a7ccd-workload-socket\") pod \"e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h\" (UID: \"2c25889c-7caa-4718-b251-5231794a7ccd\") " pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:34.488248 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.488190 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c25889c-7caa-4718-b251-5231794a7ccd-istio-podinfo\") pod \"e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h\" (UID: \"2c25889c-7caa-4718-b251-5231794a7ccd\") " pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:34.488248 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.488222 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c25889c-7caa-4718-b251-5231794a7ccd-istio-token\") pod \"e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h\" (UID: \"2c25889c-7caa-4718-b251-5231794a7ccd\") " pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:34.488248 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.488242 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c25889c-7caa-4718-b251-5231794a7ccd-istiod-ca-cert\") pod \"e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h\" (UID: \"2c25889c-7caa-4718-b251-5231794a7ccd\") " pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:34.488521 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.488311 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c25889c-7caa-4718-b251-5231794a7ccd-credential-socket\") pod \"e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h\" (UID: \"2c25889c-7caa-4718-b251-5231794a7ccd\") " pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:34.589249 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.589210 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c25889c-7caa-4718-b251-5231794a7ccd-istio-podinfo\") pod \"e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h\" (UID: \"2c25889c-7caa-4718-b251-5231794a7ccd\") " pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:34.589249 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.589249 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c25889c-7caa-4718-b251-5231794a7ccd-istio-token\") pod \"e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h\" (UID: \"2c25889c-7caa-4718-b251-5231794a7ccd\") " pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:34.589558 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.589304 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c25889c-7caa-4718-b251-5231794a7ccd-istiod-ca-cert\") pod \"e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h\" (UID: \"2c25889c-7caa-4718-b251-5231794a7ccd\") " pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:34.589558 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.589344 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c25889c-7caa-4718-b251-5231794a7ccd-credential-socket\") pod \"e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h\" (UID: \"2c25889c-7caa-4718-b251-5231794a7ccd\") " pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:34.589558 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.589467 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c25889c-7caa-4718-b251-5231794a7ccd-istio-envoy\") pod \"e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h\" (UID: \"2c25889c-7caa-4718-b251-5231794a7ccd\") " pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:34.589558 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.589504 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gtng\" (UniqueName: \"kubernetes.io/projected/2c25889c-7caa-4718-b251-5231794a7ccd-kube-api-access-4gtng\") pod \"e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h\" (UID: \"2c25889c-7caa-4718-b251-5231794a7ccd\") " pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:34.589558 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.589557 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c25889c-7caa-4718-b251-5231794a7ccd-istio-data\") pod \"e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h\" (UID: \"2c25889c-7caa-4718-b251-5231794a7ccd\") " pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:34.589838 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.589585 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c25889c-7caa-4718-b251-5231794a7ccd-workload-certs\") pod \"e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h\" (UID: \"2c25889c-7caa-4718-b251-5231794a7ccd\") " pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:34.589838 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.589619 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c25889c-7caa-4718-b251-5231794a7ccd-workload-socket\") pod \"e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h\" (UID: \"2c25889c-7caa-4718-b251-5231794a7ccd\") " pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:34.589937 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.589854 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c25889c-7caa-4718-b251-5231794a7ccd-credential-socket\") pod \"e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h\" (UID: \"2c25889c-7caa-4718-b251-5231794a7ccd\") " pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:34.590023 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.589989 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c25889c-7caa-4718-b251-5231794a7ccd-workload-socket\") pod \"e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h\" (UID: \"2c25889c-7caa-4718-b251-5231794a7ccd\") " pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:34.590084 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.590019 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c25889c-7caa-4718-b251-5231794a7ccd-workload-certs\") pod \"e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h\" (UID: \"2c25889c-7caa-4718-b251-5231794a7ccd\") " pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:34.590228 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.590206 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c25889c-7caa-4718-b251-5231794a7ccd-istiod-ca-cert\") pod \"e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h\" (UID: \"2c25889c-7caa-4718-b251-5231794a7ccd\") " pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:34.590329 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.590213 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c25889c-7caa-4718-b251-5231794a7ccd-istio-data\") pod \"e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h\" (UID: \"2c25889c-7caa-4718-b251-5231794a7ccd\") " pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:34.591748 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.591720 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c25889c-7caa-4718-b251-5231794a7ccd-istio-envoy\") pod \"e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h\" (UID: \"2c25889c-7caa-4718-b251-5231794a7ccd\") " pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:34.592080 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.592049 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c25889c-7caa-4718-b251-5231794a7ccd-istio-podinfo\") pod \"e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h\" (UID: \"2c25889c-7caa-4718-b251-5231794a7ccd\") " pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:34.598225 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.598197 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c25889c-7caa-4718-b251-5231794a7ccd-istio-token\") pod \"e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h\" (UID: \"2c25889c-7caa-4718-b251-5231794a7ccd\") " pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:34.598599 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.598583 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gtng\" (UniqueName: \"kubernetes.io/projected/2c25889c-7caa-4718-b251-5231794a7ccd-kube-api-access-4gtng\") pod \"e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h\" (UID: \"2c25889c-7caa-4718-b251-5231794a7ccd\") " pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:34.679884 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.679783 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:34.833387 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.833358 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h"] Jul 08 19:14:34.837952 ip-10-0-130-48 kubenswrapper[2570]: W0708 19:14:34.837920 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c25889c_7caa_4718_b251_5231794a7ccd.slice/crio-0e815f3a28d246e306cdc1f330fd2bba6b8c492589c174852057242f31ba76f5 WatchSource:0}: Error finding container 0e815f3a28d246e306cdc1f330fd2bba6b8c492589c174852057242f31ba76f5: Status 404 returned error can't find the container with id 0e815f3a28d246e306cdc1f330fd2bba6b8c492589c174852057242f31ba76f5 Jul 08 19:14:34.840177 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.840141 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:14:34.840272 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.840212 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:14:34.840272 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:34.840237 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:14:35.217041 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:35.216998 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" event={"ID":"2c25889c-7caa-4718-b251-5231794a7ccd","Type":"ContainerStarted","Data":"a9d2319e599474cd5c7e00dd50657a5fca9c8a4ba4b58e572e8037cb68f5541b"} Jul 08 19:14:35.217320 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:35.217065 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" event={"ID":"2c25889c-7caa-4718-b251-5231794a7ccd","Type":"ContainerStarted","Data":"0e815f3a28d246e306cdc1f330fd2bba6b8c492589c174852057242f31ba76f5"} Jul 08 19:14:35.243502 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:35.243455 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" podStartSLOduration=1.24343991 podStartE2EDuration="1.24343991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:14:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:14:35.242982115 +0000 UTC m=+1736.090256785" watchObservedRunningTime="2026-07-08 19:14:35.24343991 +0000 UTC m=+1736.090714575" Jul 08 19:14:35.680279 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:35.680228 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:35.690416 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:35.690394 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:36.223094 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:36.223064 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:36.224154 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:36.224133 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:37.400169 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:37.400138 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t"] Jul 08 19:14:37.403444 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:37.403425 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t" Jul 08 19:14:37.406461 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:37.406438 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s5q9r\"" Jul 08 19:14:37.406720 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:37.406703 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-164a1c52-serving-cert\"" Jul 08 19:14:37.436053 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:37.434520 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t"] Jul 08 19:14:37.514380 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:37.514342 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad381c9-19cf-4b59-ab69-ad9b67074ecc-openshift-service-ca\") pod \"maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t\" (UID: \"7ad381c9-19cf-4b59-ab69-ad9b67074ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t" Jul 08 19:14:37.514562 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:37.514400 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ad381c9-19cf-4b59-ab69-ad9b67074ecc-maas-api-tls\") pod \"maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t\" (UID: \"7ad381c9-19cf-4b59-ab69-ad9b67074ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t" Jul 08 19:14:37.514562 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:37.514505 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4s5d\" (UniqueName: \"kubernetes.io/projected/7ad381c9-19cf-4b59-ab69-ad9b67074ecc-kube-api-access-r4s5d\") pod \"maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t\" (UID: \"7ad381c9-19cf-4b59-ab69-ad9b67074ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t" Jul 08 19:14:37.615515 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:37.615478 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad381c9-19cf-4b59-ab69-ad9b67074ecc-openshift-service-ca\") pod \"maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t\" (UID: \"7ad381c9-19cf-4b59-ab69-ad9b67074ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t" Jul 08 19:14:37.615713 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:37.615528 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ad381c9-19cf-4b59-ab69-ad9b67074ecc-maas-api-tls\") pod \"maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t\" (UID: \"7ad381c9-19cf-4b59-ab69-ad9b67074ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t" Jul 08 19:14:37.615713 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:37.615608 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4s5d\" (UniqueName: \"kubernetes.io/projected/7ad381c9-19cf-4b59-ab69-ad9b67074ecc-kube-api-access-r4s5d\") pod \"maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t\" (UID: \"7ad381c9-19cf-4b59-ab69-ad9b67074ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t" Jul 08 19:14:37.615832 ip-10-0-130-48 kubenswrapper[2570]: E0708 19:14:37.615711 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-164a1c52-serving-cert: secret "maas-api-e2e-mt-164a1c52-serving-cert" not found Jul 08 19:14:37.615832 ip-10-0-130-48 kubenswrapper[2570]: E0708 19:14:37.615780 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ad381c9-19cf-4b59-ab69-ad9b67074ecc-maas-api-tls podName:7ad381c9-19cf-4b59-ab69-ad9b67074ecc nodeName:}" failed. No retries permitted until 2026-07-08 19:14:38.115763522 +0000 UTC m=+1738.963038166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7ad381c9-19cf-4b59-ab69-ad9b67074ecc-maas-api-tls") pod "maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t" (UID: "7ad381c9-19cf-4b59-ab69-ad9b67074ecc") : secret "maas-api-e2e-mt-164a1c52-serving-cert" not found Jul 08 19:14:37.616352 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:37.616328 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad381c9-19cf-4b59-ab69-ad9b67074ecc-openshift-service-ca\") pod \"maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t\" (UID: \"7ad381c9-19cf-4b59-ab69-ad9b67074ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t" Jul 08 19:14:37.626883 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:37.626860 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4s5d\" (UniqueName: \"kubernetes.io/projected/7ad381c9-19cf-4b59-ab69-ad9b67074ecc-kube-api-access-r4s5d\") pod \"maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t\" (UID: \"7ad381c9-19cf-4b59-ab69-ad9b67074ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t" Jul 08 19:14:38.120177 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:38.120143 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ad381c9-19cf-4b59-ab69-ad9b67074ecc-maas-api-tls\") pod \"maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t\" (UID: \"7ad381c9-19cf-4b59-ab69-ad9b67074ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t" Jul 08 19:14:38.122534 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:38.122513 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ad381c9-19cf-4b59-ab69-ad9b67074ecc-maas-api-tls\") pod \"maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t\" (UID: \"7ad381c9-19cf-4b59-ab69-ad9b67074ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t" Jul 08 19:14:38.316954 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:38.316915 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t" Jul 08 19:14:38.470340 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:38.470311 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t"] Jul 08 19:14:38.470951 ip-10-0-130-48 kubenswrapper[2570]: W0708 19:14:38.470930 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad381c9_19cf_4b59_ab69_ad9b67074ecc.slice/crio-2eed5778e840bc4872d00bd1a149ed6e6c0c1d23dfb1e59c79f8c62bffa989e0 WatchSource:0}: Error finding container 2eed5778e840bc4872d00bd1a149ed6e6c0c1d23dfb1e59c79f8c62bffa989e0: Status 404 returned error can't find the container with id 2eed5778e840bc4872d00bd1a149ed6e6c0c1d23dfb1e59c79f8c62bffa989e0 Jul 08 19:14:39.248541 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:39.248509 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t" event={"ID":"7ad381c9-19cf-4b59-ab69-ad9b67074ecc","Type":"ContainerStarted","Data":"603ca42f91cd4ef095a9da3a0c0d4b6c43e6a277f5097ff814414fee1af66445"} Jul 08 19:14:39.248716 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:39.248562 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t" event={"ID":"7ad381c9-19cf-4b59-ab69-ad9b67074ecc","Type":"ContainerStarted","Data":"2eed5778e840bc4872d00bd1a149ed6e6c0c1d23dfb1e59c79f8c62bffa989e0"} Jul 08 19:14:39.248716 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:39.248589 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t" Jul 08 19:14:39.282348 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:39.282302 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t" podStartSLOduration=1.707209727 podStartE2EDuration="2.282287772s" totalImagesPullingTime="575.078045ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:14:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:14:39.280586149 +0000 UTC m=+1740.127860805" watchObservedRunningTime="2026-07-08 19:14:39.282287772 +0000 UTC m=+1740.129562429" Jul 08 19:14:43.144956 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:43.144921 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:14:45.260057 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:45.260024 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t" Jul 08 19:14:50.211330 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:50.211289 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t"] Jul 08 19:14:50.211790 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:50.211561 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t" podUID="7ad381c9-19cf-4b59-ab69-ad9b67074ecc" containerName="maas-api" containerID="cri-o://603ca42f91cd4ef095a9da3a0c0d4b6c43e6a277f5097ff814414fee1af66445" gracePeriod=30 Jul 08 19:14:50.253970 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:50.253927 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t" podUID="7ad381c9-19cf-4b59-ab69-ad9b67074ecc" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.69:8443/health\": dial tcp 10.133.0.69:8443: connect: connection refused" Jul 08 19:14:50.479606 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:50.479577 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t" Jul 08 19:14:50.528904 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:50.528874 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4s5d\" (UniqueName: \"kubernetes.io/projected/7ad381c9-19cf-4b59-ab69-ad9b67074ecc-kube-api-access-r4s5d\") pod \"7ad381c9-19cf-4b59-ab69-ad9b67074ecc\" (UID: \"7ad381c9-19cf-4b59-ab69-ad9b67074ecc\") " Jul 08 19:14:50.529078 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:50.528975 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ad381c9-19cf-4b59-ab69-ad9b67074ecc-maas-api-tls\") pod \"7ad381c9-19cf-4b59-ab69-ad9b67074ecc\" (UID: \"7ad381c9-19cf-4b59-ab69-ad9b67074ecc\") " Jul 08 19:14:50.529078 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:50.529032 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad381c9-19cf-4b59-ab69-ad9b67074ecc-openshift-service-ca\") pod \"7ad381c9-19cf-4b59-ab69-ad9b67074ecc\" (UID: \"7ad381c9-19cf-4b59-ab69-ad9b67074ecc\") " Jul 08 19:14:50.531473 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:50.531414 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ad381c9-19cf-4b59-ab69-ad9b67074ecc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7ad381c9-19cf-4b59-ab69-ad9b67074ecc" (UID: "7ad381c9-19cf-4b59-ab69-ad9b67074ecc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:14:50.531625 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:50.531503 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ad381c9-19cf-4b59-ab69-ad9b67074ecc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7ad381c9-19cf-4b59-ab69-ad9b67074ecc" (UID: "7ad381c9-19cf-4b59-ab69-ad9b67074ecc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:14:50.532071 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:50.532042 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad381c9-19cf-4b59-ab69-ad9b67074ecc-kube-api-access-r4s5d" (OuterVolumeSpecName: "kube-api-access-r4s5d") pod "7ad381c9-19cf-4b59-ab69-ad9b67074ecc" (UID: "7ad381c9-19cf-4b59-ab69-ad9b67074ecc"). InnerVolumeSpecName "kube-api-access-r4s5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:14:50.630449 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:50.630406 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ad381c9-19cf-4b59-ab69-ad9b67074ecc-maas-api-tls\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:14:50.630449 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:50.630440 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad381c9-19cf-4b59-ab69-ad9b67074ecc-openshift-service-ca\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:14:50.630449 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:50.630451 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4s5d\" (UniqueName: \"kubernetes.io/projected/7ad381c9-19cf-4b59-ab69-ad9b67074ecc-kube-api-access-r4s5d\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:14:51.225462 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:51.225129 2570 patch_prober.go:28] interesting pod/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Jul 08 19:14:51.225462 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:51.225204 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" podUID="2c25889c-7caa-4718-b251-5231794a7ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Jul 08 19:14:51.320062 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:51.320029 2570 generic.go:356] "Generic (PLEG): container finished" podID="2c25889c-7caa-4718-b251-5231794a7ccd" containerID="a9d2319e599474cd5c7e00dd50657a5fca9c8a4ba4b58e572e8037cb68f5541b" exitCode=0 Jul 08 19:14:51.320239 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:51.320109 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" event={"ID":"2c25889c-7caa-4718-b251-5231794a7ccd","Type":"ContainerDied","Data":"a9d2319e599474cd5c7e00dd50657a5fca9c8a4ba4b58e572e8037cb68f5541b"} Jul 08 19:14:51.320614 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:51.320588 2570 scope.go:117] "RemoveContainer" containerID="a9d2319e599474cd5c7e00dd50657a5fca9c8a4ba4b58e572e8037cb68f5541b" Jul 08 19:14:51.321554 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:51.321534 2570 generic.go:356] "Generic (PLEG): container finished" podID="7ad381c9-19cf-4b59-ab69-ad9b67074ecc" containerID="603ca42f91cd4ef095a9da3a0c0d4b6c43e6a277f5097ff814414fee1af66445" exitCode=0 Jul 08 19:14:51.321648 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:51.321571 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t" event={"ID":"7ad381c9-19cf-4b59-ab69-ad9b67074ecc","Type":"ContainerDied","Data":"603ca42f91cd4ef095a9da3a0c0d4b6c43e6a277f5097ff814414fee1af66445"} Jul 08 19:14:51.321648 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:51.321608 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t" Jul 08 19:14:51.321648 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:51.321618 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t" event={"ID":"7ad381c9-19cf-4b59-ab69-ad9b67074ecc","Type":"ContainerDied","Data":"2eed5778e840bc4872d00bd1a149ed6e6c0c1d23dfb1e59c79f8c62bffa989e0"} Jul 08 19:14:51.321813 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:51.321648 2570 scope.go:117] "RemoveContainer" containerID="603ca42f91cd4ef095a9da3a0c0d4b6c43e6a277f5097ff814414fee1af66445" Jul 08 19:14:51.321813 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:51.321720 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:14:51.321891 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:51.321808 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:14:51.321891 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:51.321864 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:14:51.334417 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:51.334399 2570 scope.go:117] "RemoveContainer" containerID="603ca42f91cd4ef095a9da3a0c0d4b6c43e6a277f5097ff814414fee1af66445" Jul 08 19:14:51.334692 ip-10-0-130-48 kubenswrapper[2570]: E0708 19:14:51.334666 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"603ca42f91cd4ef095a9da3a0c0d4b6c43e6a277f5097ff814414fee1af66445\": container with ID starting with 603ca42f91cd4ef095a9da3a0c0d4b6c43e6a277f5097ff814414fee1af66445 not found: ID does not exist" containerID="603ca42f91cd4ef095a9da3a0c0d4b6c43e6a277f5097ff814414fee1af66445" Jul 08 19:14:51.334757 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:51.334692 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"603ca42f91cd4ef095a9da3a0c0d4b6c43e6a277f5097ff814414fee1af66445"} err="failed to get container status \"603ca42f91cd4ef095a9da3a0c0d4b6c43e6a277f5097ff814414fee1af66445\": rpc error: code = NotFound desc = could not find container \"603ca42f91cd4ef095a9da3a0c0d4b6c43e6a277f5097ff814414fee1af66445\": container with ID starting with 603ca42f91cd4ef095a9da3a0c0d4b6c43e6a277f5097ff814414fee1af66445 not found: ID does not exist" Jul 08 19:14:51.358461 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:51.358434 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t"] Jul 08 19:14:51.362654 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:51.362625 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-164a1c52-654cdd5b94-2xr8t"] Jul 08 19:14:51.690625 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:51.690590 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ad381c9-19cf-4b59-ab69-ad9b67074ecc" path="/var/lib/kubelet/pods/7ad381c9-19cf-4b59-ab69-ad9b67074ecc/volumes" Jul 08 19:14:52.329078 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:52.329042 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" event={"ID":"2c25889c-7caa-4718-b251-5231794a7ccd","Type":"ContainerStarted","Data":"8423890a6a553859b64622ec3dcb23bd757a4d6233f7625f9bf7d88fe19e68f9"} Jul 08 19:14:52.680374 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:52.680328 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:52.680593 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:52.680570 2570 patch_prober.go:28] interesting pod/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Jul 08 19:14:52.680652 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:52.680632 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" podUID="2c25889c-7caa-4718-b251-5231794a7ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Jul 08 19:14:53.680843 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:53.680804 2570 patch_prober.go:28] interesting pod/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Jul 08 19:14:53.681286 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:53.680869 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" podUID="2c25889c-7caa-4718-b251-5231794a7ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Jul 08 19:14:54.601731 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:54.601695 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:14:54.680784 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:54.680744 2570 patch_prober.go:28] interesting pod/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Jul 08 19:14:54.680956 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:54.680802 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" podUID="2c25889c-7caa-4718-b251-5231794a7ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Jul 08 19:14:55.680617 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:55.680583 2570 patch_prober.go:28] interesting pod/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Jul 08 19:14:55.680812 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:55.680636 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" podUID="2c25889c-7caa-4718-b251-5231794a7ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Jul 08 19:14:56.684119 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:56.684088 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:56.684558 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:56.684365 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:56.685057 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:56.685041 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:14:57.199241 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:57.199207 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h"] Jul 08 19:14:57.380478 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:57.380447 2570 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" secret="" err="secret \"e2e-mt-164a1c52-openshift-default-dockercfg-k4ncv\" not found" Jul 08 19:14:58.385783 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:14:58.385740 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" podUID="2c25889c-7caa-4718-b251-5231794a7ccd" containerName="istio-proxy" containerID="cri-o://8423890a6a553859b64622ec3dcb23bd757a4d6233f7625f9bf7d88fe19e68f9" gracePeriod=30 Jul 08 19:15:00.157247 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:00.157212 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725635-rntfb"] Jul 08 19:15:00.157687 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:00.157672 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ad381c9-19cf-4b59-ab69-ad9b67074ecc" containerName="maas-api" Jul 08 19:15:00.157738 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:00.157689 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ad381c9-19cf-4b59-ab69-ad9b67074ecc" containerName="maas-api" Jul 08 19:15:00.157780 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:00.157771 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ad381c9-19cf-4b59-ab69-ad9b67074ecc" containerName="maas-api" Jul 08 19:15:00.160420 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:00.160405 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725635-rntfb" Jul 08 19:15:00.164065 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:00.164044 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m2spm\"" Jul 08 19:15:00.170143 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:00.170126 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725635-rntfb" Jul 08 19:15:00.184430 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:00.184405 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725635-rntfb"] Jul 08 19:15:00.325022 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:00.324999 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725635-rntfb"] Jul 08 19:15:00.327015 ip-10-0-130-48 kubenswrapper[2570]: W0708 19:15:00.326987 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c67d3c5_16a4_4cd4_9711_3e92c03a9276.slice/crio-ea3e33bf29f689077a024f51380d3b9ecefb34bd1e1cb900dbdaf28e4bcd42d0 WatchSource:0}: Error finding container ea3e33bf29f689077a024f51380d3b9ecefb34bd1e1cb900dbdaf28e4bcd42d0: Status 404 returned error can't find the container with id ea3e33bf29f689077a024f51380d3b9ecefb34bd1e1cb900dbdaf28e4bcd42d0 Jul 08 19:15:00.398389 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:00.398363 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725635-rntfb" event={"ID":"2c67d3c5-16a4-4cd4-9711-3e92c03a9276","Type":"ContainerStarted","Data":"ea3e33bf29f689077a024f51380d3b9ecefb34bd1e1cb900dbdaf28e4bcd42d0"} Jul 08 19:15:01.406473 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:01.406439 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725635-rntfb" event={"ID":"2c67d3c5-16a4-4cd4-9711-3e92c03a9276","Type":"ContainerStarted","Data":"5de77b576870af9d64cac5cdcb068b8b60f5cad01df724a68c6cda18de3ddf78"} Jul 08 19:15:01.425761 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:01.425718 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725635-rntfb" podStartSLOduration=1.425703514 podStartE2EDuration="1.425703514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:15:01.424651127 +0000 UTC m=+1762.271925793" watchObservedRunningTime="2026-07-08 19:15:01.425703514 +0000 UTC m=+1762.272978180" Jul 08 19:15:03.647296 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:03.647246 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:15:03.750358 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:03.750265 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c25889c-7caa-4718-b251-5231794a7ccd-istio-token\") pod \"2c25889c-7caa-4718-b251-5231794a7ccd\" (UID: \"2c25889c-7caa-4718-b251-5231794a7ccd\") " Jul 08 19:15:03.750513 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:03.750370 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c25889c-7caa-4718-b251-5231794a7ccd-workload-certs\") pod \"2c25889c-7caa-4718-b251-5231794a7ccd\" (UID: \"2c25889c-7caa-4718-b251-5231794a7ccd\") " Jul 08 19:15:03.750513 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:03.750424 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c25889c-7caa-4718-b251-5231794a7ccd-credential-socket\") pod \"2c25889c-7caa-4718-b251-5231794a7ccd\" (UID: \"2c25889c-7caa-4718-b251-5231794a7ccd\") " Jul 08 19:15:03.750513 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:03.750474 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c25889c-7caa-4718-b251-5231794a7ccd-istio-data\") pod \"2c25889c-7caa-4718-b251-5231794a7ccd\" (UID: \"2c25889c-7caa-4718-b251-5231794a7ccd\") " Jul 08 19:15:03.750513 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:03.750503 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c25889c-7caa-4718-b251-5231794a7ccd-istiod-ca-cert\") pod \"2c25889c-7caa-4718-b251-5231794a7ccd\" (UID: \"2c25889c-7caa-4718-b251-5231794a7ccd\") " Jul 08 19:15:03.750730 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:03.750548 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c25889c-7caa-4718-b251-5231794a7ccd-workload-socket\") pod \"2c25889c-7caa-4718-b251-5231794a7ccd\" (UID: \"2c25889c-7caa-4718-b251-5231794a7ccd\") " Jul 08 19:15:03.750730 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:03.750603 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c25889c-7caa-4718-b251-5231794a7ccd-istio-podinfo\") pod \"2c25889c-7caa-4718-b251-5231794a7ccd\" (UID: \"2c25889c-7caa-4718-b251-5231794a7ccd\") " Jul 08 19:15:03.750730 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:03.750650 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c25889c-7caa-4718-b251-5231794a7ccd-istio-envoy\") pod \"2c25889c-7caa-4718-b251-5231794a7ccd\" (UID: \"2c25889c-7caa-4718-b251-5231794a7ccd\") " Jul 08 19:15:03.750730 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:03.750675 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gtng\" (UniqueName: \"kubernetes.io/projected/2c25889c-7caa-4718-b251-5231794a7ccd-kube-api-access-4gtng\") pod \"2c25889c-7caa-4718-b251-5231794a7ccd\" (UID: \"2c25889c-7caa-4718-b251-5231794a7ccd\") " Jul 08 19:15:03.750730 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:03.750704 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c25889c-7caa-4718-b251-5231794a7ccd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2c25889c-7caa-4718-b251-5231794a7ccd" (UID: "2c25889c-7caa-4718-b251-5231794a7ccd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:15:03.750984 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:03.750713 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c25889c-7caa-4718-b251-5231794a7ccd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2c25889c-7caa-4718-b251-5231794a7ccd" (UID: "2c25889c-7caa-4718-b251-5231794a7ccd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:15:03.750984 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:03.750900 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c25889c-7caa-4718-b251-5231794a7ccd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2c25889c-7caa-4718-b251-5231794a7ccd" (UID: "2c25889c-7caa-4718-b251-5231794a7ccd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:15:03.751086 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:03.751013 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c25889c-7caa-4718-b251-5231794a7ccd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2c25889c-7caa-4718-b251-5231794a7ccd" (UID: "2c25889c-7caa-4718-b251-5231794a7ccd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:15:03.751137 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:03.751109 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c25889c-7caa-4718-b251-5231794a7ccd-workload-certs\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:15:03.751137 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:03.751127 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c25889c-7caa-4718-b251-5231794a7ccd-credential-socket\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:15:03.751232 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:03.751142 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c25889c-7caa-4718-b251-5231794a7ccd-istiod-ca-cert\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:15:03.751232 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:03.751156 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c25889c-7caa-4718-b251-5231794a7ccd-workload-socket\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:15:03.751586 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:03.751539 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c25889c-7caa-4718-b251-5231794a7ccd-istio-data" (OuterVolumeSpecName: "istio-data") pod "2c25889c-7caa-4718-b251-5231794a7ccd" (UID: "2c25889c-7caa-4718-b251-5231794a7ccd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:15:03.752885 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:03.752852 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c25889c-7caa-4718-b251-5231794a7ccd-istio-token" (OuterVolumeSpecName: "istio-token") pod "2c25889c-7caa-4718-b251-5231794a7ccd" (UID: "2c25889c-7caa-4718-b251-5231794a7ccd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:15:03.753340 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:03.753310 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c25889c-7caa-4718-b251-5231794a7ccd-kube-api-access-4gtng" (OuterVolumeSpecName: "kube-api-access-4gtng") pod "2c25889c-7caa-4718-b251-5231794a7ccd" (UID: "2c25889c-7caa-4718-b251-5231794a7ccd"). InnerVolumeSpecName "kube-api-access-4gtng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:15:03.753659 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:03.753631 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c25889c-7caa-4718-b251-5231794a7ccd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2c25889c-7caa-4718-b251-5231794a7ccd" (UID: "2c25889c-7caa-4718-b251-5231794a7ccd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:15:03.753966 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:03.753936 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c25889c-7caa-4718-b251-5231794a7ccd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2c25889c-7caa-4718-b251-5231794a7ccd" (UID: "2c25889c-7caa-4718-b251-5231794a7ccd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 19:15:03.851791 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:03.851761 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c25889c-7caa-4718-b251-5231794a7ccd-istio-podinfo\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:15:03.851791 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:03.851788 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c25889c-7caa-4718-b251-5231794a7ccd-istio-envoy\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:15:03.851791 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:03.851796 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gtng\" (UniqueName: \"kubernetes.io/projected/2c25889c-7caa-4718-b251-5231794a7ccd-kube-api-access-4gtng\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:15:03.852003 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:03.851807 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c25889c-7caa-4718-b251-5231794a7ccd-istio-token\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:15:03.852003 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:03.851815 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c25889c-7caa-4718-b251-5231794a7ccd-istio-data\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:15:04.426530 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:04.426496 2570 generic.go:356] "Generic (PLEG): container finished" podID="2c25889c-7caa-4718-b251-5231794a7ccd" containerID="8423890a6a553859b64622ec3dcb23bd757a4d6233f7625f9bf7d88fe19e68f9" exitCode=0 Jul 08 19:15:04.426698 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:04.426554 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" Jul 08 19:15:04.426698 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:04.426581 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" event={"ID":"2c25889c-7caa-4718-b251-5231794a7ccd","Type":"ContainerDied","Data":"8423890a6a553859b64622ec3dcb23bd757a4d6233f7625f9bf7d88fe19e68f9"} Jul 08 19:15:04.426698 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:04.426636 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h" event={"ID":"2c25889c-7caa-4718-b251-5231794a7ccd","Type":"ContainerDied","Data":"0e815f3a28d246e306cdc1f330fd2bba6b8c492589c174852057242f31ba76f5"} Jul 08 19:15:04.426698 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:04.426657 2570 scope.go:117] "RemoveContainer" containerID="8423890a6a553859b64622ec3dcb23bd757a4d6233f7625f9bf7d88fe19e68f9" Jul 08 19:15:04.438807 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:04.438785 2570 scope.go:117] "RemoveContainer" containerID="a9d2319e599474cd5c7e00dd50657a5fca9c8a4ba4b58e572e8037cb68f5541b" Jul 08 19:15:04.448157 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:04.448137 2570 scope.go:117] "RemoveContainer" containerID="8423890a6a553859b64622ec3dcb23bd757a4d6233f7625f9bf7d88fe19e68f9" Jul 08 19:15:04.448521 ip-10-0-130-48 kubenswrapper[2570]: E0708 19:15:04.448491 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8423890a6a553859b64622ec3dcb23bd757a4d6233f7625f9bf7d88fe19e68f9\": container with ID starting with 8423890a6a553859b64622ec3dcb23bd757a4d6233f7625f9bf7d88fe19e68f9 not found: ID does not exist" containerID="8423890a6a553859b64622ec3dcb23bd757a4d6233f7625f9bf7d88fe19e68f9" Jul 08 19:15:04.448613 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:04.448525 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8423890a6a553859b64622ec3dcb23bd757a4d6233f7625f9bf7d88fe19e68f9"} err="failed to get container status \"8423890a6a553859b64622ec3dcb23bd757a4d6233f7625f9bf7d88fe19e68f9\": rpc error: code = NotFound desc = could not find container \"8423890a6a553859b64622ec3dcb23bd757a4d6233f7625f9bf7d88fe19e68f9\": container with ID starting with 8423890a6a553859b64622ec3dcb23bd757a4d6233f7625f9bf7d88fe19e68f9 not found: ID does not exist" Jul 08 19:15:04.448613 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:04.448550 2570 scope.go:117] "RemoveContainer" containerID="a9d2319e599474cd5c7e00dd50657a5fca9c8a4ba4b58e572e8037cb68f5541b" Jul 08 19:15:04.448812 ip-10-0-130-48 kubenswrapper[2570]: E0708 19:15:04.448793 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9d2319e599474cd5c7e00dd50657a5fca9c8a4ba4b58e572e8037cb68f5541b\": container with ID starting with a9d2319e599474cd5c7e00dd50657a5fca9c8a4ba4b58e572e8037cb68f5541b not found: ID does not exist" containerID="a9d2319e599474cd5c7e00dd50657a5fca9c8a4ba4b58e572e8037cb68f5541b" Jul 08 19:15:04.448871 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:04.448817 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9d2319e599474cd5c7e00dd50657a5fca9c8a4ba4b58e572e8037cb68f5541b"} err="failed to get container status \"a9d2319e599474cd5c7e00dd50657a5fca9c8a4ba4b58e572e8037cb68f5541b\": rpc error: code = NotFound desc = could not find container \"a9d2319e599474cd5c7e00dd50657a5fca9c8a4ba4b58e572e8037cb68f5541b\": container with ID starting with a9d2319e599474cd5c7e00dd50657a5fca9c8a4ba4b58e572e8037cb68f5541b not found: ID does not exist" Jul 08 19:15:04.453526 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:04.453506 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h"] Jul 08 19:15:04.466990 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:04.466967 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-164a1c52-openshift-default-56cdcb69ff-mf42h"] Jul 08 19:15:05.690791 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:05.690755 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c25889c-7caa-4718-b251-5231794a7ccd" path="/var/lib/kubelet/pods/2c25889c-7caa-4718-b251-5231794a7ccd/volumes" Jul 08 19:15:17.811638 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.811585 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh"] Jul 08 19:15:17.812403 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.812382 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c25889c-7caa-4718-b251-5231794a7ccd" containerName="istio-proxy" Jul 08 19:15:17.812403 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.812404 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c25889c-7caa-4718-b251-5231794a7ccd" containerName="istio-proxy" Jul 08 19:15:17.812568 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.812433 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c25889c-7caa-4718-b251-5231794a7ccd" containerName="istio-proxy" Jul 08 19:15:17.812568 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.812441 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c25889c-7caa-4718-b251-5231794a7ccd" containerName="istio-proxy" Jul 08 19:15:17.812568 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.812533 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c25889c-7caa-4718-b251-5231794a7ccd" containerName="istio-proxy" Jul 08 19:15:17.812568 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.812546 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c25889c-7caa-4718-b251-5231794a7ccd" containerName="istio-proxy" Jul 08 19:15:17.815804 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.815782 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:17.819923 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.819848 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a04868c7-openshift-default-dockercfg-pjpfs\"" Jul 08 19:15:17.833791 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.833710 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh"] Jul 08 19:15:17.874810 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.874775 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a801b8-1ca7-4069-8785-067b5ab74744-credential-socket\") pod \"e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh\" (UID: \"c0a801b8-1ca7-4069-8785-067b5ab74744\") " pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:17.874810 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.874809 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0a801b8-1ca7-4069-8785-067b5ab74744-istio-podinfo\") pod \"e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh\" (UID: \"c0a801b8-1ca7-4069-8785-067b5ab74744\") " pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:17.875010 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.874829 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0a801b8-1ca7-4069-8785-067b5ab74744-istio-envoy\") pod \"e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh\" (UID: \"c0a801b8-1ca7-4069-8785-067b5ab74744\") " pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:17.875010 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.874843 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0a801b8-1ca7-4069-8785-067b5ab74744-istio-data\") pod \"e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh\" (UID: \"c0a801b8-1ca7-4069-8785-067b5ab74744\") " pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:17.875010 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.874896 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0a801b8-1ca7-4069-8785-067b5ab74744-workload-certs\") pod \"e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh\" (UID: \"c0a801b8-1ca7-4069-8785-067b5ab74744\") " pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:17.875010 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.874944 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvppl\" (UniqueName: \"kubernetes.io/projected/c0a801b8-1ca7-4069-8785-067b5ab74744-kube-api-access-rvppl\") pod \"e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh\" (UID: \"c0a801b8-1ca7-4069-8785-067b5ab74744\") " pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:17.875010 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.874975 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0a801b8-1ca7-4069-8785-067b5ab74744-istiod-ca-cert\") pod \"e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh\" (UID: \"c0a801b8-1ca7-4069-8785-067b5ab74744\") " pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:17.875243 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.875019 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a801b8-1ca7-4069-8785-067b5ab74744-workload-socket\") pod \"e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh\" (UID: \"c0a801b8-1ca7-4069-8785-067b5ab74744\") " pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:17.875243 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.875098 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0a801b8-1ca7-4069-8785-067b5ab74744-istio-token\") pod \"e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh\" (UID: \"c0a801b8-1ca7-4069-8785-067b5ab74744\") " pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:17.976323 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.976287 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0a801b8-1ca7-4069-8785-067b5ab74744-istiod-ca-cert\") pod \"e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh\" (UID: \"c0a801b8-1ca7-4069-8785-067b5ab74744\") " pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:17.976494 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.976358 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a801b8-1ca7-4069-8785-067b5ab74744-workload-socket\") pod \"e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh\" (UID: \"c0a801b8-1ca7-4069-8785-067b5ab74744\") " pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:17.976494 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.976401 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0a801b8-1ca7-4069-8785-067b5ab74744-istio-token\") pod \"e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh\" (UID: \"c0a801b8-1ca7-4069-8785-067b5ab74744\") " pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:17.976494 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.976434 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a801b8-1ca7-4069-8785-067b5ab74744-credential-socket\") pod \"e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh\" (UID: \"c0a801b8-1ca7-4069-8785-067b5ab74744\") " pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:17.976494 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.976450 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0a801b8-1ca7-4069-8785-067b5ab74744-istio-podinfo\") pod \"e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh\" (UID: \"c0a801b8-1ca7-4069-8785-067b5ab74744\") " pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:17.976494 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.976477 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0a801b8-1ca7-4069-8785-067b5ab74744-istio-envoy\") pod \"e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh\" (UID: \"c0a801b8-1ca7-4069-8785-067b5ab74744\") " pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:17.976778 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.976500 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0a801b8-1ca7-4069-8785-067b5ab74744-istio-data\") pod \"e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh\" (UID: \"c0a801b8-1ca7-4069-8785-067b5ab74744\") " pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:17.976778 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.976538 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0a801b8-1ca7-4069-8785-067b5ab74744-workload-certs\") pod \"e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh\" (UID: \"c0a801b8-1ca7-4069-8785-067b5ab74744\") " pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:17.976778 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.976573 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvppl\" (UniqueName: \"kubernetes.io/projected/c0a801b8-1ca7-4069-8785-067b5ab74744-kube-api-access-rvppl\") pod \"e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh\" (UID: \"c0a801b8-1ca7-4069-8785-067b5ab74744\") " pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:17.976944 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.976788 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a801b8-1ca7-4069-8785-067b5ab74744-workload-socket\") pod \"e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh\" (UID: \"c0a801b8-1ca7-4069-8785-067b5ab74744\") " pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:17.977046 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.977025 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0a801b8-1ca7-4069-8785-067b5ab74744-istio-data\") pod \"e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh\" (UID: \"c0a801b8-1ca7-4069-8785-067b5ab74744\") " pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:17.977113 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.977080 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0a801b8-1ca7-4069-8785-067b5ab74744-workload-certs\") pod \"e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh\" (UID: \"c0a801b8-1ca7-4069-8785-067b5ab74744\") " pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:17.977113 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.977097 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0a801b8-1ca7-4069-8785-067b5ab74744-istiod-ca-cert\") pod \"e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh\" (UID: \"c0a801b8-1ca7-4069-8785-067b5ab74744\") " pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:17.977222 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.977125 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a801b8-1ca7-4069-8785-067b5ab74744-credential-socket\") pod \"e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh\" (UID: \"c0a801b8-1ca7-4069-8785-067b5ab74744\") " pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:17.978780 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.978753 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0a801b8-1ca7-4069-8785-067b5ab74744-istio-envoy\") pod \"e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh\" (UID: \"c0a801b8-1ca7-4069-8785-067b5ab74744\") " pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:17.978976 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.978958 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0a801b8-1ca7-4069-8785-067b5ab74744-istio-podinfo\") pod \"e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh\" (UID: \"c0a801b8-1ca7-4069-8785-067b5ab74744\") " pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:17.992178 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.992118 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvppl\" (UniqueName: \"kubernetes.io/projected/c0a801b8-1ca7-4069-8785-067b5ab74744-kube-api-access-rvppl\") pod \"e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh\" (UID: \"c0a801b8-1ca7-4069-8785-067b5ab74744\") " pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:17.993189 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:17.993169 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0a801b8-1ca7-4069-8785-067b5ab74744-istio-token\") pod \"e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh\" (UID: \"c0a801b8-1ca7-4069-8785-067b5ab74744\") " pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:18.127999 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:18.127968 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:18.279431 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:18.279393 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh"] Jul 08 19:15:18.280447 ip-10-0-130-48 kubenswrapper[2570]: W0708 19:15:18.280422 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0a801b8_1ca7_4069_8785_067b5ab74744.slice/crio-2072bec266ad932ef55ffef07ea87f1227cd658c1d0f4a194345855acc6ad4ac WatchSource:0}: Error finding container 2072bec266ad932ef55ffef07ea87f1227cd658c1d0f4a194345855acc6ad4ac: Status 404 returned error can't find the container with id 2072bec266ad932ef55ffef07ea87f1227cd658c1d0f4a194345855acc6ad4ac Jul 08 19:15:18.282701 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:18.282663 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:15:18.282778 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:18.282743 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:15:18.282813 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:18.282781 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:15:18.512771 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:18.512663 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" event={"ID":"c0a801b8-1ca7-4069-8785-067b5ab74744","Type":"ContainerStarted","Data":"4dee33d35901de35f22cabd0ba5a2f86e98aac1732d15d3bbe9b1f41041aabb7"} Jul 08 19:15:18.513032 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:18.513009 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" event={"ID":"c0a801b8-1ca7-4069-8785-067b5ab74744","Type":"ContainerStarted","Data":"2072bec266ad932ef55ffef07ea87f1227cd658c1d0f4a194345855acc6ad4ac"} Jul 08 19:15:18.536010 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:18.535958 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" podStartSLOduration=1.535939341 podStartE2EDuration="1.535939341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:15:18.53523748 +0000 UTC m=+1779.382512151" watchObservedRunningTime="2026-07-08 19:15:18.535939341 +0000 UTC m=+1779.383214008" Jul 08 19:15:19.129182 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:19.129145 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:19.135085 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:19.135055 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:19.518468 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:19.518390 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:19.519384 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:19.519366 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:20.561618 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:20.561585 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a04868c7-586887d776-67z54"] Jul 08 19:15:20.565380 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:20.565352 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a04868c7-586887d776-67z54" Jul 08 19:15:20.567968 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:20.567945 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a04868c7-serving-cert\"" Jul 08 19:15:20.568626 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:20.568608 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s5q9r\"" Jul 08 19:15:20.578460 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:20.578436 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a04868c7-586887d776-67z54"] Jul 08 19:15:20.702982 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:20.702948 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb703cde-46ca-4232-b395-e7b0bd3f0efd-maas-api-tls\") pod \"maas-api-e2e-mt-a04868c7-586887d776-67z54\" (UID: \"cb703cde-46ca-4232-b395-e7b0bd3f0efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a04868c7-586887d776-67z54" Jul 08 19:15:20.703191 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:20.702996 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n29m\" (UniqueName: \"kubernetes.io/projected/cb703cde-46ca-4232-b395-e7b0bd3f0efd-kube-api-access-6n29m\") pod \"maas-api-e2e-mt-a04868c7-586887d776-67z54\" (UID: \"cb703cde-46ca-4232-b395-e7b0bd3f0efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a04868c7-586887d776-67z54" Jul 08 19:15:20.703301 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:20.703203 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb703cde-46ca-4232-b395-e7b0bd3f0efd-openshift-service-ca\") pod \"maas-api-e2e-mt-a04868c7-586887d776-67z54\" (UID: \"cb703cde-46ca-4232-b395-e7b0bd3f0efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a04868c7-586887d776-67z54" Jul 08 19:15:20.804773 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:20.804735 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n29m\" (UniqueName: \"kubernetes.io/projected/cb703cde-46ca-4232-b395-e7b0bd3f0efd-kube-api-access-6n29m\") pod \"maas-api-e2e-mt-a04868c7-586887d776-67z54\" (UID: \"cb703cde-46ca-4232-b395-e7b0bd3f0efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a04868c7-586887d776-67z54" Jul 08 19:15:20.804997 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:20.804890 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb703cde-46ca-4232-b395-e7b0bd3f0efd-openshift-service-ca\") pod \"maas-api-e2e-mt-a04868c7-586887d776-67z54\" (UID: \"cb703cde-46ca-4232-b395-e7b0bd3f0efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a04868c7-586887d776-67z54" Jul 08 19:15:20.804997 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:20.804963 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb703cde-46ca-4232-b395-e7b0bd3f0efd-maas-api-tls\") pod \"maas-api-e2e-mt-a04868c7-586887d776-67z54\" (UID: \"cb703cde-46ca-4232-b395-e7b0bd3f0efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a04868c7-586887d776-67z54" Jul 08 19:15:20.805134 ip-10-0-130-48 kubenswrapper[2570]: E0708 19:15:20.805103 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-a04868c7-serving-cert: secret "maas-api-e2e-mt-a04868c7-serving-cert" not found Jul 08 19:15:20.805191 ip-10-0-130-48 kubenswrapper[2570]: E0708 19:15:20.805176 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb703cde-46ca-4232-b395-e7b0bd3f0efd-maas-api-tls podName:cb703cde-46ca-4232-b395-e7b0bd3f0efd nodeName:}" failed. No retries permitted until 2026-07-08 19:15:21.305153617 +0000 UTC m=+1782.152428260 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cb703cde-46ca-4232-b395-e7b0bd3f0efd-maas-api-tls") pod "maas-api-e2e-mt-a04868c7-586887d776-67z54" (UID: "cb703cde-46ca-4232-b395-e7b0bd3f0efd") : secret "maas-api-e2e-mt-a04868c7-serving-cert" not found Jul 08 19:15:20.805746 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:20.805716 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb703cde-46ca-4232-b395-e7b0bd3f0efd-openshift-service-ca\") pod \"maas-api-e2e-mt-a04868c7-586887d776-67z54\" (UID: \"cb703cde-46ca-4232-b395-e7b0bd3f0efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a04868c7-586887d776-67z54" Jul 08 19:15:20.814038 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:20.813980 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n29m\" (UniqueName: \"kubernetes.io/projected/cb703cde-46ca-4232-b395-e7b0bd3f0efd-kube-api-access-6n29m\") pod \"maas-api-e2e-mt-a04868c7-586887d776-67z54\" (UID: \"cb703cde-46ca-4232-b395-e7b0bd3f0efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a04868c7-586887d776-67z54" Jul 08 19:15:21.309104 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:21.309060 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb703cde-46ca-4232-b395-e7b0bd3f0efd-maas-api-tls\") pod \"maas-api-e2e-mt-a04868c7-586887d776-67z54\" (UID: \"cb703cde-46ca-4232-b395-e7b0bd3f0efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a04868c7-586887d776-67z54" Jul 08 19:15:21.309300 ip-10-0-130-48 kubenswrapper[2570]: E0708 19:15:21.309220 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-a04868c7-serving-cert: secret "maas-api-e2e-mt-a04868c7-serving-cert" not found Jul 08 19:15:21.309350 ip-10-0-130-48 kubenswrapper[2570]: E0708 19:15:21.309312 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb703cde-46ca-4232-b395-e7b0bd3f0efd-maas-api-tls podName:cb703cde-46ca-4232-b395-e7b0bd3f0efd nodeName:}" failed. No retries permitted until 2026-07-08 19:15:22.309295088 +0000 UTC m=+1783.156569732 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cb703cde-46ca-4232-b395-e7b0bd3f0efd-maas-api-tls") pod "maas-api-e2e-mt-a04868c7-586887d776-67z54" (UID: "cb703cde-46ca-4232-b395-e7b0bd3f0efd") : secret "maas-api-e2e-mt-a04868c7-serving-cert" not found Jul 08 19:15:22.318972 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:22.318936 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb703cde-46ca-4232-b395-e7b0bd3f0efd-maas-api-tls\") pod \"maas-api-e2e-mt-a04868c7-586887d776-67z54\" (UID: \"cb703cde-46ca-4232-b395-e7b0bd3f0efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a04868c7-586887d776-67z54" Jul 08 19:15:22.321392 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:22.321369 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb703cde-46ca-4232-b395-e7b0bd3f0efd-maas-api-tls\") pod \"maas-api-e2e-mt-a04868c7-586887d776-67z54\" (UID: \"cb703cde-46ca-4232-b395-e7b0bd3f0efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a04868c7-586887d776-67z54" Jul 08 19:15:22.377883 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:22.377849 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a04868c7-586887d776-67z54" Jul 08 19:15:22.523245 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:22.523217 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a04868c7-586887d776-67z54"] Jul 08 19:15:22.524826 ip-10-0-130-48 kubenswrapper[2570]: W0708 19:15:22.524799 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb703cde_46ca_4232_b395_e7b0bd3f0efd.slice/crio-c0f7111132806089a43e37b7aba459225ce69a89fd1645927a4db6dedc5ec7fe WatchSource:0}: Error finding container c0f7111132806089a43e37b7aba459225ce69a89fd1645927a4db6dedc5ec7fe: Status 404 returned error can't find the container with id c0f7111132806089a43e37b7aba459225ce69a89fd1645927a4db6dedc5ec7fe Jul 08 19:15:22.538559 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:22.538525 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a04868c7-586887d776-67z54" event={"ID":"cb703cde-46ca-4232-b395-e7b0bd3f0efd","Type":"ContainerStarted","Data":"c0f7111132806089a43e37b7aba459225ce69a89fd1645927a4db6dedc5ec7fe"} Jul 08 19:15:23.548283 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:23.548220 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a04868c7-586887d776-67z54" event={"ID":"cb703cde-46ca-4232-b395-e7b0bd3f0efd","Type":"ContainerStarted","Data":"c93109d9e6b085c640b8709aff4dc540b94fd1d29e91634f4003d2368829b022"} Jul 08 19:15:23.548698 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:23.548303 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a04868c7-586887d776-67z54" Jul 08 19:15:23.567264 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:23.567214 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a04868c7-586887d776-67z54" podStartSLOduration=3.194589412 podStartE2EDuration="3.567198742s" totalImagesPullingTime="372.60933ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:15:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:15:23.565594209 +0000 UTC m=+1784.412868883" watchObservedRunningTime="2026-07-08 19:15:23.567198742 +0000 UTC m=+1784.414473407" Jul 08 19:15:26.574683 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:26.574651 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:15:29.562080 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:29.562043 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a04868c7-586887d776-67z54" Jul 08 19:15:34.519973 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:34.519862 2570 patch_prober.go:28] interesting pod/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Jul 08 19:15:34.519973 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:34.519923 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" podUID="c0a801b8-1ca7-4069-8785-067b5ab74744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Jul 08 19:15:34.619467 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:34.619429 2570 generic.go:356] "Generic (PLEG): container finished" podID="c0a801b8-1ca7-4069-8785-067b5ab74744" containerID="4dee33d35901de35f22cabd0ba5a2f86e98aac1732d15d3bbe9b1f41041aabb7" exitCode=0 Jul 08 19:15:34.619653 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:34.619506 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" event={"ID":"c0a801b8-1ca7-4069-8785-067b5ab74744","Type":"ContainerDied","Data":"4dee33d35901de35f22cabd0ba5a2f86e98aac1732d15d3bbe9b1f41041aabb7"} Jul 08 19:15:34.619982 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:34.619961 2570 scope.go:117] "RemoveContainer" containerID="4dee33d35901de35f22cabd0ba5a2f86e98aac1732d15d3bbe9b1f41041aabb7" Jul 08 19:15:34.621013 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:34.620976 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:15:34.621133 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:34.621066 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:15:34.621198 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:34.621140 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:15:35.626644 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:35.626612 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" event={"ID":"c0a801b8-1ca7-4069-8785-067b5ab74744","Type":"ContainerStarted","Data":"85b91808ca1c865215087c687918066d97e019f54fa703f8b315508f9aaf268c"} Jul 08 19:15:36.128603 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:36.128559 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:36.128804 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:36.128779 2570 patch_prober.go:28] interesting pod/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Jul 08 19:15:36.128857 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:36.128826 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" podUID="c0a801b8-1ca7-4069-8785-067b5ab74744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Jul 08 19:15:37.128438 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:37.128406 2570 patch_prober.go:28] interesting pod/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Jul 08 19:15:37.128834 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:37.128468 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" podUID="c0a801b8-1ca7-4069-8785-067b5ab74744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Jul 08 19:15:38.129493 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:38.129443 2570 patch_prober.go:28] interesting pod/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Jul 08 19:15:38.130125 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:38.130061 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" podUID="c0a801b8-1ca7-4069-8785-067b5ab74744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Jul 08 19:15:39.128591 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:39.128556 2570 patch_prober.go:28] interesting pod/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Jul 08 19:15:39.128792 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:39.128612 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" podUID="c0a801b8-1ca7-4069-8785-067b5ab74744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Jul 08 19:15:40.133585 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:40.133553 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:40.134020 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:40.133877 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:40.134802 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:40.134784 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:15:41.800329 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:41.800290 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:15:57.692066 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:15:57.692034 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:16:00.011486 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:00.011451 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725635-rntfb"] Jul 08 19:16:00.011895 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:00.011660 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725635-rntfb" podUID="2c67d3c5-16a4-4cd4-9711-3e92c03a9276" containerName="cleanup" containerID="cri-o://5de77b576870af9d64cac5cdcb068b8b60f5cad01df724a68c6cda18de3ddf78" gracePeriod=30 Jul 08 19:16:03.210315 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:03.210281 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a04868c7-586887d776-67z54"] Jul 08 19:16:03.210760 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:03.210577 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a04868c7-586887d776-67z54" podUID="cb703cde-46ca-4232-b395-e7b0bd3f0efd" containerName="maas-api" containerID="cri-o://c93109d9e6b085c640b8709aff4dc540b94fd1d29e91634f4003d2368829b022" gracePeriod=30 Jul 08 19:16:03.467528 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:03.467473 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a04868c7-586887d776-67z54" Jul 08 19:16:03.632156 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:03.632118 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6n29m\" (UniqueName: \"kubernetes.io/projected/cb703cde-46ca-4232-b395-e7b0bd3f0efd-kube-api-access-6n29m\") pod \"cb703cde-46ca-4232-b395-e7b0bd3f0efd\" (UID: \"cb703cde-46ca-4232-b395-e7b0bd3f0efd\") " Jul 08 19:16:03.632375 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:03.632227 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb703cde-46ca-4232-b395-e7b0bd3f0efd-maas-api-tls\") pod \"cb703cde-46ca-4232-b395-e7b0bd3f0efd\" (UID: \"cb703cde-46ca-4232-b395-e7b0bd3f0efd\") " Jul 08 19:16:03.632451 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:03.632385 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb703cde-46ca-4232-b395-e7b0bd3f0efd-openshift-service-ca\") pod \"cb703cde-46ca-4232-b395-e7b0bd3f0efd\" (UID: \"cb703cde-46ca-4232-b395-e7b0bd3f0efd\") " Jul 08 19:16:03.632789 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:03.632749 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb703cde-46ca-4232-b395-e7b0bd3f0efd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cb703cde-46ca-4232-b395-e7b0bd3f0efd" (UID: "cb703cde-46ca-4232-b395-e7b0bd3f0efd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:16:03.634541 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:03.634517 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb703cde-46ca-4232-b395-e7b0bd3f0efd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cb703cde-46ca-4232-b395-e7b0bd3f0efd" (UID: "cb703cde-46ca-4232-b395-e7b0bd3f0efd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:16:03.634825 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:03.634802 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb703cde-46ca-4232-b395-e7b0bd3f0efd-kube-api-access-6n29m" (OuterVolumeSpecName: "kube-api-access-6n29m") pod "cb703cde-46ca-4232-b395-e7b0bd3f0efd" (UID: "cb703cde-46ca-4232-b395-e7b0bd3f0efd"). InnerVolumeSpecName "kube-api-access-6n29m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:16:03.734358 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:03.734285 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb703cde-46ca-4232-b395-e7b0bd3f0efd-openshift-service-ca\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:16:03.734358 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:03.734314 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n29m\" (UniqueName: \"kubernetes.io/projected/cb703cde-46ca-4232-b395-e7b0bd3f0efd-kube-api-access-6n29m\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:16:03.734358 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:03.734326 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb703cde-46ca-4232-b395-e7b0bd3f0efd-maas-api-tls\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:16:03.795862 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:03.795827 2570 generic.go:356] "Generic (PLEG): container finished" podID="cb703cde-46ca-4232-b395-e7b0bd3f0efd" containerID="c93109d9e6b085c640b8709aff4dc540b94fd1d29e91634f4003d2368829b022" exitCode=0 Jul 08 19:16:03.796013 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:03.795870 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a04868c7-586887d776-67z54" Jul 08 19:16:03.796013 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:03.795908 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a04868c7-586887d776-67z54" event={"ID":"cb703cde-46ca-4232-b395-e7b0bd3f0efd","Type":"ContainerDied","Data":"c93109d9e6b085c640b8709aff4dc540b94fd1d29e91634f4003d2368829b022"} Jul 08 19:16:03.796013 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:03.795960 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a04868c7-586887d776-67z54" event={"ID":"cb703cde-46ca-4232-b395-e7b0bd3f0efd","Type":"ContainerDied","Data":"c0f7111132806089a43e37b7aba459225ce69a89fd1645927a4db6dedc5ec7fe"} Jul 08 19:16:03.796013 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:03.795982 2570 scope.go:117] "RemoveContainer" containerID="c93109d9e6b085c640b8709aff4dc540b94fd1d29e91634f4003d2368829b022" Jul 08 19:16:03.805195 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:03.805178 2570 scope.go:117] "RemoveContainer" containerID="c93109d9e6b085c640b8709aff4dc540b94fd1d29e91634f4003d2368829b022" Jul 08 19:16:03.805498 ip-10-0-130-48 kubenswrapper[2570]: E0708 19:16:03.805479 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c93109d9e6b085c640b8709aff4dc540b94fd1d29e91634f4003d2368829b022\": container with ID starting with c93109d9e6b085c640b8709aff4dc540b94fd1d29e91634f4003d2368829b022 not found: ID does not exist" containerID="c93109d9e6b085c640b8709aff4dc540b94fd1d29e91634f4003d2368829b022" Jul 08 19:16:03.805555 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:03.805504 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c93109d9e6b085c640b8709aff4dc540b94fd1d29e91634f4003d2368829b022"} err="failed to get container status \"c93109d9e6b085c640b8709aff4dc540b94fd1d29e91634f4003d2368829b022\": rpc error: code = NotFound desc = could not find container \"c93109d9e6b085c640b8709aff4dc540b94fd1d29e91634f4003d2368829b022\": container with ID starting with c93109d9e6b085c640b8709aff4dc540b94fd1d29e91634f4003d2368829b022 not found: ID does not exist" Jul 08 19:16:03.814524 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:03.814503 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a04868c7-586887d776-67z54"] Jul 08 19:16:03.819423 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:03.819403 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a04868c7-586887d776-67z54"] Jul 08 19:16:05.690143 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:05.690107 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb703cde-46ca-4232-b395-e7b0bd3f0efd" path="/var/lib/kubelet/pods/cb703cde-46ca-4232-b395-e7b0bd3f0efd/volumes" Jul 08 19:16:07.894678 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:07.894645 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:16:18.193560 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:18.193527 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:16:30.192232 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:30.192206 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725635-rntfb" Jul 08 19:16:30.961018 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:30.960982 2570 generic.go:356] "Generic (PLEG): container finished" podID="2c67d3c5-16a4-4cd4-9711-3e92c03a9276" containerID="5de77b576870af9d64cac5cdcb068b8b60f5cad01df724a68c6cda18de3ddf78" exitCode=137 Jul 08 19:16:30.961194 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:30.961026 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725635-rntfb" event={"ID":"2c67d3c5-16a4-4cd4-9711-3e92c03a9276","Type":"ContainerDied","Data":"5de77b576870af9d64cac5cdcb068b8b60f5cad01df724a68c6cda18de3ddf78"} Jul 08 19:16:30.961194 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:30.961030 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725635-rntfb" Jul 08 19:16:30.961194 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:30.961065 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725635-rntfb" event={"ID":"2c67d3c5-16a4-4cd4-9711-3e92c03a9276","Type":"ContainerDied","Data":"ea3e33bf29f689077a024f51380d3b9ecefb34bd1e1cb900dbdaf28e4bcd42d0"} Jul 08 19:16:30.961194 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:30.961086 2570 scope.go:117] "RemoveContainer" containerID="5de77b576870af9d64cac5cdcb068b8b60f5cad01df724a68c6cda18de3ddf78" Jul 08 19:16:30.970569 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:30.970551 2570 scope.go:117] "RemoveContainer" containerID="5de77b576870af9d64cac5cdcb068b8b60f5cad01df724a68c6cda18de3ddf78" Jul 08 19:16:30.970810 ip-10-0-130-48 kubenswrapper[2570]: E0708 19:16:30.970793 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5de77b576870af9d64cac5cdcb068b8b60f5cad01df724a68c6cda18de3ddf78\": container with ID starting with 5de77b576870af9d64cac5cdcb068b8b60f5cad01df724a68c6cda18de3ddf78 not found: ID does not exist" containerID="5de77b576870af9d64cac5cdcb068b8b60f5cad01df724a68c6cda18de3ddf78" Jul 08 19:16:30.970856 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:30.970818 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5de77b576870af9d64cac5cdcb068b8b60f5cad01df724a68c6cda18de3ddf78"} err="failed to get container status \"5de77b576870af9d64cac5cdcb068b8b60f5cad01df724a68c6cda18de3ddf78\": rpc error: code = NotFound desc = could not find container \"5de77b576870af9d64cac5cdcb068b8b60f5cad01df724a68c6cda18de3ddf78\": container with ID starting with 5de77b576870af9d64cac5cdcb068b8b60f5cad01df724a68c6cda18de3ddf78 not found: ID does not exist" Jul 08 19:16:30.982394 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:30.982369 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725635-rntfb"] Jul 08 19:16:30.985247 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:30.985228 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725635-rntfb"] Jul 08 19:16:31.690069 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:31.690037 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c67d3c5-16a4-4cd4-9711-3e92c03a9276" path="/var/lib/kubelet/pods/2c67d3c5-16a4-4cd4-9711-3e92c03a9276/volumes" Jul 08 19:16:53.400548 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:53.400494 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:16:56.162833 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:56.162798 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh"] Jul 08 19:16:56.163223 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:16:56.163056 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" podUID="c0a801b8-1ca7-4069-8785-067b5ab74744" containerName="istio-proxy" containerID="cri-o://85b91808ca1c865215087c687918066d97e019f54fa703f8b315508f9aaf268c" gracePeriod=30 Jul 08 19:17:01.422435 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:01.422407 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:17:01.569182 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:01.569099 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a801b8-1ca7-4069-8785-067b5ab74744-credential-socket\") pod \"c0a801b8-1ca7-4069-8785-067b5ab74744\" (UID: \"c0a801b8-1ca7-4069-8785-067b5ab74744\") " Jul 08 19:17:01.569182 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:01.569137 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a801b8-1ca7-4069-8785-067b5ab74744-workload-socket\") pod \"c0a801b8-1ca7-4069-8785-067b5ab74744\" (UID: \"c0a801b8-1ca7-4069-8785-067b5ab74744\") " Jul 08 19:17:01.569182 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:01.569162 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0a801b8-1ca7-4069-8785-067b5ab74744-istiod-ca-cert\") pod \"c0a801b8-1ca7-4069-8785-067b5ab74744\" (UID: \"c0a801b8-1ca7-4069-8785-067b5ab74744\") " Jul 08 19:17:01.569499 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:01.569187 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0a801b8-1ca7-4069-8785-067b5ab74744-istio-token\") pod \"c0a801b8-1ca7-4069-8785-067b5ab74744\" (UID: \"c0a801b8-1ca7-4069-8785-067b5ab74744\") " Jul 08 19:17:01.569499 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:01.569238 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0a801b8-1ca7-4069-8785-067b5ab74744-workload-certs\") pod \"c0a801b8-1ca7-4069-8785-067b5ab74744\" (UID: \"c0a801b8-1ca7-4069-8785-067b5ab74744\") " Jul 08 19:17:01.569499 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:01.569301 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0a801b8-1ca7-4069-8785-067b5ab74744-istio-podinfo\") pod \"c0a801b8-1ca7-4069-8785-067b5ab74744\" (UID: \"c0a801b8-1ca7-4069-8785-067b5ab74744\") " Jul 08 19:17:01.569499 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:01.569363 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0a801b8-1ca7-4069-8785-067b5ab74744-istio-data\") pod \"c0a801b8-1ca7-4069-8785-067b5ab74744\" (UID: \"c0a801b8-1ca7-4069-8785-067b5ab74744\") " Jul 08 19:17:01.569499 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:01.569450 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0a801b8-1ca7-4069-8785-067b5ab74744-istio-envoy\") pod \"c0a801b8-1ca7-4069-8785-067b5ab74744\" (UID: \"c0a801b8-1ca7-4069-8785-067b5ab74744\") " Jul 08 19:17:01.569499 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:01.569497 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvppl\" (UniqueName: \"kubernetes.io/projected/c0a801b8-1ca7-4069-8785-067b5ab74744-kube-api-access-rvppl\") pod \"c0a801b8-1ca7-4069-8785-067b5ab74744\" (UID: \"c0a801b8-1ca7-4069-8785-067b5ab74744\") " Jul 08 19:17:01.569810 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:01.569465 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a801b8-1ca7-4069-8785-067b5ab74744-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c0a801b8-1ca7-4069-8785-067b5ab74744" (UID: "c0a801b8-1ca7-4069-8785-067b5ab74744"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:17:01.569810 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:01.569524 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a801b8-1ca7-4069-8785-067b5ab74744-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c0a801b8-1ca7-4069-8785-067b5ab74744" (UID: "c0a801b8-1ca7-4069-8785-067b5ab74744"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:17:01.569810 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:01.569551 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0a801b8-1ca7-4069-8785-067b5ab74744-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c0a801b8-1ca7-4069-8785-067b5ab74744" (UID: "c0a801b8-1ca7-4069-8785-067b5ab74744"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:17:01.569810 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:01.569700 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a801b8-1ca7-4069-8785-067b5ab74744-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c0a801b8-1ca7-4069-8785-067b5ab74744" (UID: "c0a801b8-1ca7-4069-8785-067b5ab74744"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:17:01.570013 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:01.569840 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0a801b8-1ca7-4069-8785-067b5ab74744-workload-certs\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:17:01.570013 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:01.569859 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a801b8-1ca7-4069-8785-067b5ab74744-credential-socket\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:17:01.570013 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:01.569869 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a801b8-1ca7-4069-8785-067b5ab74744-workload-socket\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:17:01.570013 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:01.569879 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0a801b8-1ca7-4069-8785-067b5ab74744-istiod-ca-cert\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:17:01.570485 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:01.570460 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a801b8-1ca7-4069-8785-067b5ab74744-istio-data" (OuterVolumeSpecName: "istio-data") pod "c0a801b8-1ca7-4069-8785-067b5ab74744" (UID: "c0a801b8-1ca7-4069-8785-067b5ab74744"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:17:01.571738 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:01.571711 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0a801b8-1ca7-4069-8785-067b5ab74744-kube-api-access-rvppl" (OuterVolumeSpecName: "kube-api-access-rvppl") pod "c0a801b8-1ca7-4069-8785-067b5ab74744" (UID: "c0a801b8-1ca7-4069-8785-067b5ab74744"). InnerVolumeSpecName "kube-api-access-rvppl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:17:01.571806 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:01.571711 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a801b8-1ca7-4069-8785-067b5ab74744-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c0a801b8-1ca7-4069-8785-067b5ab74744" (UID: "c0a801b8-1ca7-4069-8785-067b5ab74744"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:17:01.571879 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:01.571860 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0a801b8-1ca7-4069-8785-067b5ab74744-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c0a801b8-1ca7-4069-8785-067b5ab74744" (UID: "c0a801b8-1ca7-4069-8785-067b5ab74744"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 19:17:01.571991 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:01.571971 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0a801b8-1ca7-4069-8785-067b5ab74744-istio-token" (OuterVolumeSpecName: "istio-token") pod "c0a801b8-1ca7-4069-8785-067b5ab74744" (UID: "c0a801b8-1ca7-4069-8785-067b5ab74744"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:17:01.671171 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:01.671139 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0a801b8-1ca7-4069-8785-067b5ab74744-istio-envoy\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:17:01.671171 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:01.671167 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvppl\" (UniqueName: \"kubernetes.io/projected/c0a801b8-1ca7-4069-8785-067b5ab74744-kube-api-access-rvppl\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:17:01.671171 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:01.671178 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0a801b8-1ca7-4069-8785-067b5ab74744-istio-token\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:17:01.671437 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:01.671187 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0a801b8-1ca7-4069-8785-067b5ab74744-istio-podinfo\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:17:01.671437 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:01.671195 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0a801b8-1ca7-4069-8785-067b5ab74744-istio-data\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:17:02.139058 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:02.139020 2570 generic.go:356] "Generic (PLEG): container finished" podID="c0a801b8-1ca7-4069-8785-067b5ab74744" containerID="85b91808ca1c865215087c687918066d97e019f54fa703f8b315508f9aaf268c" exitCode=0 Jul 08 19:17:02.139234 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:02.139073 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" Jul 08 19:17:02.139234 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:02.139070 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" event={"ID":"c0a801b8-1ca7-4069-8785-067b5ab74744","Type":"ContainerDied","Data":"85b91808ca1c865215087c687918066d97e019f54fa703f8b315508f9aaf268c"} Jul 08 19:17:02.139234 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:02.139186 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh" event={"ID":"c0a801b8-1ca7-4069-8785-067b5ab74744","Type":"ContainerDied","Data":"2072bec266ad932ef55ffef07ea87f1227cd658c1d0f4a194345855acc6ad4ac"} Jul 08 19:17:02.139234 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:02.139214 2570 scope.go:117] "RemoveContainer" containerID="85b91808ca1c865215087c687918066d97e019f54fa703f8b315508f9aaf268c" Jul 08 19:17:02.148360 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:02.148324 2570 scope.go:117] "RemoveContainer" containerID="4dee33d35901de35f22cabd0ba5a2f86e98aac1732d15d3bbe9b1f41041aabb7" Jul 08 19:17:02.156215 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:02.156143 2570 scope.go:117] "RemoveContainer" containerID="85b91808ca1c865215087c687918066d97e019f54fa703f8b315508f9aaf268c" Jul 08 19:17:02.156518 ip-10-0-130-48 kubenswrapper[2570]: E0708 19:17:02.156496 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85b91808ca1c865215087c687918066d97e019f54fa703f8b315508f9aaf268c\": container with ID starting with 85b91808ca1c865215087c687918066d97e019f54fa703f8b315508f9aaf268c not found: ID does not exist" containerID="85b91808ca1c865215087c687918066d97e019f54fa703f8b315508f9aaf268c" Jul 08 19:17:02.156620 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:02.156525 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85b91808ca1c865215087c687918066d97e019f54fa703f8b315508f9aaf268c"} err="failed to get container status \"85b91808ca1c865215087c687918066d97e019f54fa703f8b315508f9aaf268c\": rpc error: code = NotFound desc = could not find container \"85b91808ca1c865215087c687918066d97e019f54fa703f8b315508f9aaf268c\": container with ID starting with 85b91808ca1c865215087c687918066d97e019f54fa703f8b315508f9aaf268c not found: ID does not exist" Jul 08 19:17:02.156620 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:02.156546 2570 scope.go:117] "RemoveContainer" containerID="4dee33d35901de35f22cabd0ba5a2f86e98aac1732d15d3bbe9b1f41041aabb7" Jul 08 19:17:02.156825 ip-10-0-130-48 kubenswrapper[2570]: E0708 19:17:02.156804 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dee33d35901de35f22cabd0ba5a2f86e98aac1732d15d3bbe9b1f41041aabb7\": container with ID starting with 4dee33d35901de35f22cabd0ba5a2f86e98aac1732d15d3bbe9b1f41041aabb7 not found: ID does not exist" containerID="4dee33d35901de35f22cabd0ba5a2f86e98aac1732d15d3bbe9b1f41041aabb7" Jul 08 19:17:02.156895 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:02.156830 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dee33d35901de35f22cabd0ba5a2f86e98aac1732d15d3bbe9b1f41041aabb7"} err="failed to get container status \"4dee33d35901de35f22cabd0ba5a2f86e98aac1732d15d3bbe9b1f41041aabb7\": rpc error: code = NotFound desc = could not find container \"4dee33d35901de35f22cabd0ba5a2f86e98aac1732d15d3bbe9b1f41041aabb7\": container with ID starting with 4dee33d35901de35f22cabd0ba5a2f86e98aac1732d15d3bbe9b1f41041aabb7 not found: ID does not exist" Jul 08 19:17:02.175202 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:02.175180 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh"] Jul 08 19:17:02.190824 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:02.190805 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a04868c7-openshift-default-757c546bc-tv5nh"] Jul 08 19:17:03.691009 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:03.690964 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0a801b8-1ca7-4069-8785-067b5ab74744" path="/var/lib/kubelet/pods/c0a801b8-1ca7-4069-8785-067b5ab74744/volumes" Jul 08 19:17:07.064958 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:07.064926 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:17:16.505094 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:16.505055 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:17:52.380154 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.380124 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm"] Jul 08 19:17:52.380583 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.380568 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0a801b8-1ca7-4069-8785-067b5ab74744" containerName="istio-proxy" Jul 08 19:17:52.380628 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.380584 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0a801b8-1ca7-4069-8785-067b5ab74744" containerName="istio-proxy" Jul 08 19:17:52.380628 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.380594 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c67d3c5-16a4-4cd4-9711-3e92c03a9276" containerName="cleanup" Jul 08 19:17:52.380628 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.380599 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c67d3c5-16a4-4cd4-9711-3e92c03a9276" containerName="cleanup" Jul 08 19:17:52.380628 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.380607 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0a801b8-1ca7-4069-8785-067b5ab74744" containerName="istio-proxy" Jul 08 19:17:52.380628 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.380613 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0a801b8-1ca7-4069-8785-067b5ab74744" containerName="istio-proxy" Jul 08 19:17:52.380628 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.380621 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb703cde-46ca-4232-b395-e7b0bd3f0efd" containerName="maas-api" Jul 08 19:17:52.380628 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.380626 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb703cde-46ca-4232-b395-e7b0bd3f0efd" containerName="maas-api" Jul 08 19:17:52.380856 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.380702 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c67d3c5-16a4-4cd4-9711-3e92c03a9276" containerName="cleanup" Jul 08 19:17:52.380856 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.380712 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb703cde-46ca-4232-b395-e7b0bd3f0efd" containerName="maas-api" Jul 08 19:17:52.380856 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.380719 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0a801b8-1ca7-4069-8785-067b5ab74744" containerName="istio-proxy" Jul 08 19:17:52.380856 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.380724 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0a801b8-1ca7-4069-8785-067b5ab74744" containerName="istio-proxy" Jul 08 19:17:52.383855 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.383839 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:17:52.387030 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.387002 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b274ce-openshift-default-dockercfg-6tsmj\"" Jul 08 19:17:52.398439 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.398414 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm"] Jul 08 19:17:52.509405 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.509370 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-istio-envoy\") pod \"e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm\" (UID: \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\") " pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:17:52.509593 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.509425 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqwk9\" (UniqueName: \"kubernetes.io/projected/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-kube-api-access-qqwk9\") pod \"e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm\" (UID: \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\") " pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:17:52.509593 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.509567 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-credential-socket\") pod \"e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm\" (UID: \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\") " pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:17:52.509703 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.509610 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-istio-data\") pod \"e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm\" (UID: \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\") " pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:17:52.509703 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.509650 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-workload-certs\") pod \"e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm\" (UID: \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\") " pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:17:52.509703 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.509681 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-workload-socket\") pod \"e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm\" (UID: \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\") " pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:17:52.509703 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.509700 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-istio-podinfo\") pod \"e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm\" (UID: \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\") " pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:17:52.509858 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.509722 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-istiod-ca-cert\") pod \"e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm\" (UID: \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\") " pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:17:52.509858 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.509748 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-istio-token\") pod \"e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm\" (UID: \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\") " pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:17:52.610825 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.610791 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-credential-socket\") pod \"e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm\" (UID: \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\") " pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:17:52.610825 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.610830 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-istio-data\") pod \"e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm\" (UID: \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\") " pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:17:52.611024 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.610853 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-workload-certs\") pod \"e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm\" (UID: \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\") " pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:17:52.611024 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.610893 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-workload-socket\") pod \"e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm\" (UID: \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\") " pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:17:52.611024 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.610927 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-istio-podinfo\") pod \"e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm\" (UID: \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\") " pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:17:52.611024 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.610955 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-istiod-ca-cert\") pod \"e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm\" (UID: \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\") " pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:17:52.611024 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.610990 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-istio-token\") pod \"e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm\" (UID: \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\") " pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:17:52.611327 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.611028 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-istio-envoy\") pod \"e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm\" (UID: \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\") " pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:17:52.611327 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.611071 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqwk9\" (UniqueName: \"kubernetes.io/projected/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-kube-api-access-qqwk9\") pod \"e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm\" (UID: \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\") " pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:17:52.611327 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.611293 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-credential-socket\") pod \"e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm\" (UID: \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\") " pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:17:52.611496 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.611325 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-workload-certs\") pod \"e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm\" (UID: \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\") " pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:17:52.611496 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.611388 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-workload-socket\") pod \"e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm\" (UID: \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\") " pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:17:52.611602 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.611574 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-istio-data\") pod \"e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm\" (UID: \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\") " pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:17:52.611789 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.611767 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-istiod-ca-cert\") pod \"e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm\" (UID: \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\") " pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:17:52.613494 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.613474 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-istio-envoy\") pod \"e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm\" (UID: \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\") " pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:17:52.613543 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.613474 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-istio-podinfo\") pod \"e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm\" (UID: \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\") " pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:17:52.622711 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.622687 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-istio-token\") pod \"e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm\" (UID: \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\") " pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:17:52.623752 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.623731 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqwk9\" (UniqueName: \"kubernetes.io/projected/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-kube-api-access-qqwk9\") pod \"e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm\" (UID: \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\") " pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:17:52.695300 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.695209 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:17:52.835110 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.835074 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm"] Jul 08 19:17:52.838326 ip-10-0-130-48 kubenswrapper[2570]: W0708 19:17:52.838293 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3d5682_c7a5_49c0_96e3_f4a1b8f6d8f0.slice/crio-aa2a71106e29105dd872c3fbc20b8fb3348a1ac18eced6aae3394e0a94f21b0e WatchSource:0}: Error finding container aa2a71106e29105dd872c3fbc20b8fb3348a1ac18eced6aae3394e0a94f21b0e: Status 404 returned error can't find the container with id aa2a71106e29105dd872c3fbc20b8fb3348a1ac18eced6aae3394e0a94f21b0e Jul 08 19:17:52.840741 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.840707 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:17:52.840837 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.840771 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:17:52.840837 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:52.840802 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:17:53.429192 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:53.429156 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" event={"ID":"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0","Type":"ContainerStarted","Data":"78860cdfe3f1aa6aa23d15dce711b32244d429324866859df2d874e69a7ad2c5"} Jul 08 19:17:53.429608 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:53.429216 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" event={"ID":"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0","Type":"ContainerStarted","Data":"aa2a71106e29105dd872c3fbc20b8fb3348a1ac18eced6aae3394e0a94f21b0e"} Jul 08 19:17:53.461446 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:53.461392 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" podStartSLOduration=1.461376492 podStartE2EDuration="1.461376492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:17:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:17:53.458567755 +0000 UTC m=+1934.305842420" watchObservedRunningTime="2026-07-08 19:17:53.461376492 +0000 UTC m=+1934.308651149" Jul 08 19:17:53.696097 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:53.696022 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:17:53.700930 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:53.700905 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:17:54.436087 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:54.436049 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:17:54.437171 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:54.437148 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:17:55.500542 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:55.500507 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b274ce-84868b8996-vnx67"] Jul 08 19:17:55.504900 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:55.504877 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b274ce-84868b8996-vnx67" Jul 08 19:17:55.507564 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:55.507533 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b274ce-serving-cert\"" Jul 08 19:17:55.507697 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:55.507575 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s5q9r\"" Jul 08 19:17:55.513806 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:55.513783 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b274ce-84868b8996-vnx67"] Jul 08 19:17:55.638149 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:55.638105 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/636d7ac7-8b4d-40d8-8f60-09141a6f3cd4-maas-api-tls\") pod \"maas-api-e2e-inf-a-b274ce-84868b8996-vnx67\" (UID: \"636d7ac7-8b4d-40d8-8f60-09141a6f3cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b274ce-84868b8996-vnx67" Jul 08 19:17:55.638369 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:55.638169 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4w9f\" (UniqueName: \"kubernetes.io/projected/636d7ac7-8b4d-40d8-8f60-09141a6f3cd4-kube-api-access-j4w9f\") pod \"maas-api-e2e-inf-a-b274ce-84868b8996-vnx67\" (UID: \"636d7ac7-8b4d-40d8-8f60-09141a6f3cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b274ce-84868b8996-vnx67" Jul 08 19:17:55.638369 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:55.638199 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/636d7ac7-8b4d-40d8-8f60-09141a6f3cd4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b274ce-84868b8996-vnx67\" (UID: \"636d7ac7-8b4d-40d8-8f60-09141a6f3cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b274ce-84868b8996-vnx67" Jul 08 19:17:55.739531 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:55.739489 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/636d7ac7-8b4d-40d8-8f60-09141a6f3cd4-maas-api-tls\") pod \"maas-api-e2e-inf-a-b274ce-84868b8996-vnx67\" (UID: \"636d7ac7-8b4d-40d8-8f60-09141a6f3cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b274ce-84868b8996-vnx67" Jul 08 19:17:55.739715 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:55.739549 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4w9f\" (UniqueName: \"kubernetes.io/projected/636d7ac7-8b4d-40d8-8f60-09141a6f3cd4-kube-api-access-j4w9f\") pod \"maas-api-e2e-inf-a-b274ce-84868b8996-vnx67\" (UID: \"636d7ac7-8b4d-40d8-8f60-09141a6f3cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b274ce-84868b8996-vnx67" Jul 08 19:17:55.739715 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:55.739588 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/636d7ac7-8b4d-40d8-8f60-09141a6f3cd4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b274ce-84868b8996-vnx67\" (UID: \"636d7ac7-8b4d-40d8-8f60-09141a6f3cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b274ce-84868b8996-vnx67" Jul 08 19:17:55.739715 ip-10-0-130-48 kubenswrapper[2570]: E0708 19:17:55.739646 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-b274ce-serving-cert: secret "maas-api-e2e-inf-a-b274ce-serving-cert" not found Jul 08 19:17:55.739715 ip-10-0-130-48 kubenswrapper[2570]: E0708 19:17:55.739714 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/636d7ac7-8b4d-40d8-8f60-09141a6f3cd4-maas-api-tls podName:636d7ac7-8b4d-40d8-8f60-09141a6f3cd4 nodeName:}" failed. No retries permitted until 2026-07-08 19:17:56.239691455 +0000 UTC m=+1937.086966102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/636d7ac7-8b4d-40d8-8f60-09141a6f3cd4-maas-api-tls") pod "maas-api-e2e-inf-a-b274ce-84868b8996-vnx67" (UID: "636d7ac7-8b4d-40d8-8f60-09141a6f3cd4") : secret "maas-api-e2e-inf-a-b274ce-serving-cert" not found Jul 08 19:17:55.740291 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:55.740271 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/636d7ac7-8b4d-40d8-8f60-09141a6f3cd4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b274ce-84868b8996-vnx67\" (UID: \"636d7ac7-8b4d-40d8-8f60-09141a6f3cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b274ce-84868b8996-vnx67" Jul 08 19:17:55.781786 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:55.781725 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4w9f\" (UniqueName: \"kubernetes.io/projected/636d7ac7-8b4d-40d8-8f60-09141a6f3cd4-kube-api-access-j4w9f\") pod \"maas-api-e2e-inf-a-b274ce-84868b8996-vnx67\" (UID: \"636d7ac7-8b4d-40d8-8f60-09141a6f3cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b274ce-84868b8996-vnx67" Jul 08 19:17:56.244989 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:56.244956 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/636d7ac7-8b4d-40d8-8f60-09141a6f3cd4-maas-api-tls\") pod \"maas-api-e2e-inf-a-b274ce-84868b8996-vnx67\" (UID: \"636d7ac7-8b4d-40d8-8f60-09141a6f3cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b274ce-84868b8996-vnx67" Jul 08 19:17:56.247363 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:56.247339 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/636d7ac7-8b4d-40d8-8f60-09141a6f3cd4-maas-api-tls\") pod \"maas-api-e2e-inf-a-b274ce-84868b8996-vnx67\" (UID: \"636d7ac7-8b4d-40d8-8f60-09141a6f3cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b274ce-84868b8996-vnx67" Jul 08 19:17:56.419491 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:56.419459 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b274ce-84868b8996-vnx67" Jul 08 19:17:56.560564 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:56.560524 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b274ce-84868b8996-vnx67"] Jul 08 19:17:56.563630 ip-10-0-130-48 kubenswrapper[2570]: W0708 19:17:56.563588 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636d7ac7_8b4d_40d8_8f60_09141a6f3cd4.slice/crio-26583c2985c55e3b429ad610b4e864a19e46380d41acbe4db188522d535576b9 WatchSource:0}: Error finding container 26583c2985c55e3b429ad610b4e864a19e46380d41acbe4db188522d535576b9: Status 404 returned error can't find the container with id 26583c2985c55e3b429ad610b4e864a19e46380d41acbe4db188522d535576b9 Jul 08 19:17:56.565057 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:56.565035 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 19:17:57.459156 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:57.459116 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b274ce-84868b8996-vnx67" event={"ID":"636d7ac7-8b4d-40d8-8f60-09141a6f3cd4","Type":"ContainerStarted","Data":"2fd6ad2776c793b1f947308927f2082aa77df47bb547be9b37ff242ac77b67f9"} Jul 08 19:17:57.459403 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:57.459188 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b274ce-84868b8996-vnx67" event={"ID":"636d7ac7-8b4d-40d8-8f60-09141a6f3cd4","Type":"ContainerStarted","Data":"26583c2985c55e3b429ad610b4e864a19e46380d41acbe4db188522d535576b9"} Jul 08 19:17:57.459403 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:57.459353 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b274ce-84868b8996-vnx67" Jul 08 19:17:57.476371 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:17:57.476326 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b274ce-84868b8996-vnx67" podStartSLOduration=2.041869794 podStartE2EDuration="2.476312575s" totalImagesPullingTime="434.442781ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:17:57.474863767 +0000 UTC m=+1938.322138433" watchObservedRunningTime="2026-07-08 19:17:57.476312575 +0000 UTC m=+1938.323587255" Jul 08 19:18:03.470750 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:03.470721 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b274ce-84868b8996-vnx67" Jul 08 19:18:03.794632 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:03.794556 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:18:10.479386 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:10.479344 2570 patch_prober.go:28] interesting pod/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 08 19:18:10.479775 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:10.479397 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" podUID="da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 08 19:18:12.547320 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:12.547288 2570 generic.go:356] "Generic (PLEG): container finished" podID="da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0" containerID="78860cdfe3f1aa6aa23d15dce711b32244d429324866859df2d874e69a7ad2c5" exitCode=0 Jul 08 19:18:12.547822 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:12.547340 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" event={"ID":"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0","Type":"ContainerDied","Data":"78860cdfe3f1aa6aa23d15dce711b32244d429324866859df2d874e69a7ad2c5"} Jul 08 19:18:12.547822 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:12.547755 2570 scope.go:117] "RemoveContainer" containerID="78860cdfe3f1aa6aa23d15dce711b32244d429324866859df2d874e69a7ad2c5" Jul 08 19:18:12.549044 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:12.549007 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:18:12.549178 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:12.549107 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:18:12.549178 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:12.549157 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:18:13.559757 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:13.559726 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" event={"ID":"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0","Type":"ContainerStarted","Data":"fbda5de873151560e0a8cece6b7f6d3f8694f947c930b3f792a03494062d8e59"} Jul 08 19:18:13.695380 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:13.695352 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:18:13.695572 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:13.695552 2570 patch_prober.go:28] interesting pod/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Jul 08 19:18:13.695624 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:13.695599 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" podUID="da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Jul 08 19:18:14.696599 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:14.696558 2570 patch_prober.go:28] interesting pod/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Jul 08 19:18:14.697079 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:14.696617 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" podUID="da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Jul 08 19:18:15.695726 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:15.695693 2570 patch_prober.go:28] interesting pod/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Jul 08 19:18:15.695914 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:15.695741 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" podUID="da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Jul 08 19:18:16.696096 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:16.696061 2570 patch_prober.go:28] interesting pod/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Jul 08 19:18:16.696512 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:16.696118 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" podUID="da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Jul 08 19:18:17.699625 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:17.699598 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:18:17.700097 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:17.699999 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:18:17.700762 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:17.700739 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:18:19.307075 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:19.307043 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:18:39.495539 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:39.495457 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:18:39.875164 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:39.875132 2570 scope.go:117] "RemoveContainer" containerID="c707fa62166c3d60bec4ea4910ca2339b259bfe8388ade938e2b7e2451b4d7fb" Jul 08 19:18:58.858186 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:58.858147 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b274ce-84868b8996-vnx67"] Jul 08 19:18:58.858685 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:58.858439 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b274ce-84868b8996-vnx67" podUID="636d7ac7-8b4d-40d8-8f60-09141a6f3cd4" containerName="maas-api" containerID="cri-o://2fd6ad2776c793b1f947308927f2082aa77df47bb547be9b37ff242ac77b67f9" gracePeriod=30 Jul 08 19:18:59.117144 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:59.117120 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b274ce-84868b8996-vnx67" Jul 08 19:18:59.251526 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:59.251490 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/636d7ac7-8b4d-40d8-8f60-09141a6f3cd4-openshift-service-ca\") pod \"636d7ac7-8b4d-40d8-8f60-09141a6f3cd4\" (UID: \"636d7ac7-8b4d-40d8-8f60-09141a6f3cd4\") " Jul 08 19:18:59.251694 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:59.251561 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/636d7ac7-8b4d-40d8-8f60-09141a6f3cd4-maas-api-tls\") pod \"636d7ac7-8b4d-40d8-8f60-09141a6f3cd4\" (UID: \"636d7ac7-8b4d-40d8-8f60-09141a6f3cd4\") " Jul 08 19:18:59.251694 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:59.251635 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4w9f\" (UniqueName: \"kubernetes.io/projected/636d7ac7-8b4d-40d8-8f60-09141a6f3cd4-kube-api-access-j4w9f\") pod \"636d7ac7-8b4d-40d8-8f60-09141a6f3cd4\" (UID: \"636d7ac7-8b4d-40d8-8f60-09141a6f3cd4\") " Jul 08 19:18:59.251933 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:59.251902 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/636d7ac7-8b4d-40d8-8f60-09141a6f3cd4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "636d7ac7-8b4d-40d8-8f60-09141a6f3cd4" (UID: "636d7ac7-8b4d-40d8-8f60-09141a6f3cd4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:18:59.253692 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:59.253660 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/636d7ac7-8b4d-40d8-8f60-09141a6f3cd4-kube-api-access-j4w9f" (OuterVolumeSpecName: "kube-api-access-j4w9f") pod "636d7ac7-8b4d-40d8-8f60-09141a6f3cd4" (UID: "636d7ac7-8b4d-40d8-8f60-09141a6f3cd4"). InnerVolumeSpecName "kube-api-access-j4w9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:18:59.253806 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:59.253688 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/636d7ac7-8b4d-40d8-8f60-09141a6f3cd4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "636d7ac7-8b4d-40d8-8f60-09141a6f3cd4" (UID: "636d7ac7-8b4d-40d8-8f60-09141a6f3cd4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:18:59.352351 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:59.352318 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4w9f\" (UniqueName: \"kubernetes.io/projected/636d7ac7-8b4d-40d8-8f60-09141a6f3cd4-kube-api-access-j4w9f\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:18:59.352351 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:59.352349 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/636d7ac7-8b4d-40d8-8f60-09141a6f3cd4-openshift-service-ca\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:18:59.352558 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:59.352363 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/636d7ac7-8b4d-40d8-8f60-09141a6f3cd4-maas-api-tls\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:18:59.833736 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:59.833698 2570 generic.go:356] "Generic (PLEG): container finished" podID="636d7ac7-8b4d-40d8-8f60-09141a6f3cd4" containerID="2fd6ad2776c793b1f947308927f2082aa77df47bb547be9b37ff242ac77b67f9" exitCode=0 Jul 08 19:18:59.833905 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:59.833766 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b274ce-84868b8996-vnx67" Jul 08 19:18:59.833905 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:59.833801 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b274ce-84868b8996-vnx67" event={"ID":"636d7ac7-8b4d-40d8-8f60-09141a6f3cd4","Type":"ContainerDied","Data":"2fd6ad2776c793b1f947308927f2082aa77df47bb547be9b37ff242ac77b67f9"} Jul 08 19:18:59.833905 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:59.833874 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b274ce-84868b8996-vnx67" event={"ID":"636d7ac7-8b4d-40d8-8f60-09141a6f3cd4","Type":"ContainerDied","Data":"26583c2985c55e3b429ad610b4e864a19e46380d41acbe4db188522d535576b9"} Jul 08 19:18:59.834052 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:59.833911 2570 scope.go:117] "RemoveContainer" containerID="2fd6ad2776c793b1f947308927f2082aa77df47bb547be9b37ff242ac77b67f9" Jul 08 19:18:59.843541 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:59.843516 2570 scope.go:117] "RemoveContainer" containerID="2fd6ad2776c793b1f947308927f2082aa77df47bb547be9b37ff242ac77b67f9" Jul 08 19:18:59.843843 ip-10-0-130-48 kubenswrapper[2570]: E0708 19:18:59.843818 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fd6ad2776c793b1f947308927f2082aa77df47bb547be9b37ff242ac77b67f9\": container with ID starting with 2fd6ad2776c793b1f947308927f2082aa77df47bb547be9b37ff242ac77b67f9 not found: ID does not exist" containerID="2fd6ad2776c793b1f947308927f2082aa77df47bb547be9b37ff242ac77b67f9" Jul 08 19:18:59.843942 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:59.843852 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fd6ad2776c793b1f947308927f2082aa77df47bb547be9b37ff242ac77b67f9"} err="failed to get container status \"2fd6ad2776c793b1f947308927f2082aa77df47bb547be9b37ff242ac77b67f9\": rpc error: code = NotFound desc = could not find container \"2fd6ad2776c793b1f947308927f2082aa77df47bb547be9b37ff242ac77b67f9\": container with ID starting with 2fd6ad2776c793b1f947308927f2082aa77df47bb547be9b37ff242ac77b67f9 not found: ID does not exist" Jul 08 19:18:59.852452 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:59.852428 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b274ce-84868b8996-vnx67"] Jul 08 19:18:59.858849 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:18:59.858825 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b274ce-84868b8996-vnx67"] Jul 08 19:19:01.690568 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:01.690534 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="636d7ac7-8b4d-40d8-8f60-09141a6f3cd4" path="/var/lib/kubelet/pods/636d7ac7-8b4d-40d8-8f60-09141a6f3cd4/volumes" Jul 08 19:19:04.089625 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:04.089583 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:19:41.043668 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:41.043634 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm"] Jul 08 19:19:41.044181 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:41.043838 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" podUID="da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0" containerName="istio-proxy" containerID="cri-o://fbda5de873151560e0a8cece6b7f6d3f8694f947c930b3f792a03494062d8e59" gracePeriod=30 Jul 08 19:19:45.297042 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:45.297007 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:19:46.299013 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:46.298990 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:19:46.378128 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:46.378102 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqwk9\" (UniqueName: \"kubernetes.io/projected/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-kube-api-access-qqwk9\") pod \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\" (UID: \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\") " Jul 08 19:19:46.378312 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:46.378135 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-istio-data\") pod \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\" (UID: \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\") " Jul 08 19:19:46.378312 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:46.378156 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-workload-certs\") pod \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\" (UID: \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\") " Jul 08 19:19:46.378312 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:46.378175 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-istio-podinfo\") pod \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\" (UID: \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\") " Jul 08 19:19:46.378312 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:46.378205 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-istiod-ca-cert\") pod \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\" (UID: \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\") " Jul 08 19:19:46.378312 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:46.378280 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-credential-socket\") pod \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\" (UID: \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\") " Jul 08 19:19:46.378584 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:46.378332 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-istio-token\") pod \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\" (UID: \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\") " Jul 08 19:19:46.378584 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:46.378435 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-istio-envoy\") pod \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\" (UID: \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\") " Jul 08 19:19:46.378584 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:46.378479 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-workload-socket\") pod \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\" (UID: \"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0\") " Jul 08 19:19:46.378584 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:46.378499 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0" (UID: "da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:19:46.378785 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:46.378641 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0" (UID: "da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:19:46.378859 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:46.378812 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-workload-certs\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:19:46.378920 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:46.378868 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-credential-socket\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:19:46.378920 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:46.378870 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0" (UID: "da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:19:46.379084 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:46.379047 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0" (UID: "da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:19:46.379203 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:46.379179 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-istio-data" (OuterVolumeSpecName: "istio-data") pod "da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0" (UID: "da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:19:46.380540 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:46.380518 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0" (UID: "da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 19:19:46.380642 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:46.380535 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-kube-api-access-qqwk9" (OuterVolumeSpecName: "kube-api-access-qqwk9") pod "da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0" (UID: "da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0"). InnerVolumeSpecName "kube-api-access-qqwk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:19:46.380875 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:46.380853 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0" (UID: "da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:19:46.380951 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:46.380901 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-istio-token" (OuterVolumeSpecName: "istio-token") pod "da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0" (UID: "da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:19:46.479318 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:46.479295 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-istio-envoy\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:19:46.479318 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:46.479316 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-workload-socket\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:19:46.479318 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:46.479325 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqwk9\" (UniqueName: \"kubernetes.io/projected/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-kube-api-access-qqwk9\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:19:46.479509 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:46.479336 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-istio-data\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:19:46.479509 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:46.479345 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-istio-podinfo\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:19:46.479509 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:46.479353 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-istiod-ca-cert\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:19:46.479509 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:46.479360 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0-istio-token\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:19:47.104916 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:47.104880 2570 generic.go:356] "Generic (PLEG): container finished" podID="da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0" containerID="fbda5de873151560e0a8cece6b7f6d3f8694f947c930b3f792a03494062d8e59" exitCode=0 Jul 08 19:19:47.105122 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:47.104946 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" Jul 08 19:19:47.105122 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:47.104963 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" event={"ID":"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0","Type":"ContainerDied","Data":"fbda5de873151560e0a8cece6b7f6d3f8694f947c930b3f792a03494062d8e59"} Jul 08 19:19:47.105122 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:47.105013 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm" event={"ID":"da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0","Type":"ContainerDied","Data":"aa2a71106e29105dd872c3fbc20b8fb3348a1ac18eced6aae3394e0a94f21b0e"} Jul 08 19:19:47.105122 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:47.105039 2570 scope.go:117] "RemoveContainer" containerID="fbda5de873151560e0a8cece6b7f6d3f8694f947c930b3f792a03494062d8e59" Jul 08 19:19:47.114993 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:47.114975 2570 scope.go:117] "RemoveContainer" containerID="78860cdfe3f1aa6aa23d15dce711b32244d429324866859df2d874e69a7ad2c5" Jul 08 19:19:47.123978 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:47.123958 2570 scope.go:117] "RemoveContainer" containerID="fbda5de873151560e0a8cece6b7f6d3f8694f947c930b3f792a03494062d8e59" Jul 08 19:19:47.124238 ip-10-0-130-48 kubenswrapper[2570]: E0708 19:19:47.124220 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbda5de873151560e0a8cece6b7f6d3f8694f947c930b3f792a03494062d8e59\": container with ID starting with fbda5de873151560e0a8cece6b7f6d3f8694f947c930b3f792a03494062d8e59 not found: ID does not exist" containerID="fbda5de873151560e0a8cece6b7f6d3f8694f947c930b3f792a03494062d8e59" Jul 08 19:19:47.124307 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:47.124246 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbda5de873151560e0a8cece6b7f6d3f8694f947c930b3f792a03494062d8e59"} err="failed to get container status \"fbda5de873151560e0a8cece6b7f6d3f8694f947c930b3f792a03494062d8e59\": rpc error: code = NotFound desc = could not find container \"fbda5de873151560e0a8cece6b7f6d3f8694f947c930b3f792a03494062d8e59\": container with ID starting with fbda5de873151560e0a8cece6b7f6d3f8694f947c930b3f792a03494062d8e59 not found: ID does not exist" Jul 08 19:19:47.124307 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:47.124283 2570 scope.go:117] "RemoveContainer" containerID="78860cdfe3f1aa6aa23d15dce711b32244d429324866859df2d874e69a7ad2c5" Jul 08 19:19:47.124510 ip-10-0-130-48 kubenswrapper[2570]: E0708 19:19:47.124489 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78860cdfe3f1aa6aa23d15dce711b32244d429324866859df2d874e69a7ad2c5\": container with ID starting with 78860cdfe3f1aa6aa23d15dce711b32244d429324866859df2d874e69a7ad2c5 not found: ID does not exist" containerID="78860cdfe3f1aa6aa23d15dce711b32244d429324866859df2d874e69a7ad2c5" Jul 08 19:19:47.124568 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:47.124521 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78860cdfe3f1aa6aa23d15dce711b32244d429324866859df2d874e69a7ad2c5"} err="failed to get container status \"78860cdfe3f1aa6aa23d15dce711b32244d429324866859df2d874e69a7ad2c5\": rpc error: code = NotFound desc = could not find container \"78860cdfe3f1aa6aa23d15dce711b32244d429324866859df2d874e69a7ad2c5\": container with ID starting with 78860cdfe3f1aa6aa23d15dce711b32244d429324866859df2d874e69a7ad2c5 not found: ID does not exist" Jul 08 19:19:47.131112 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:47.130837 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm"] Jul 08 19:19:47.133771 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:47.133746 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b274ce-openshift-default-6556c55d7f-zgzmm"] Jul 08 19:19:47.690035 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:47.690003 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0" path="/var/lib/kubelet/pods/da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0/volumes" Jul 08 19:19:49.317560 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.317522 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f"] Jul 08 19:19:49.317976 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.317962 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="636d7ac7-8b4d-40d8-8f60-09141a6f3cd4" containerName="maas-api" Jul 08 19:19:49.317976 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.317977 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="636d7ac7-8b4d-40d8-8f60-09141a6f3cd4" containerName="maas-api" Jul 08 19:19:49.318049 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.317989 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0" containerName="istio-proxy" Jul 08 19:19:49.318049 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.317994 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0" containerName="istio-proxy" Jul 08 19:19:49.318049 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.318005 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0" containerName="istio-proxy" Jul 08 19:19:49.318049 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.318011 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0" containerName="istio-proxy" Jul 08 19:19:49.318177 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.318068 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0" containerName="istio-proxy" Jul 08 19:19:49.318177 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.318076 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="636d7ac7-8b4d-40d8-8f60-09141a6f3cd4" containerName="maas-api" Jul 08 19:19:49.318177 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.318085 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="da3d5682-c7a5-49c0-96e3-f4a1b8f6d8f0" containerName="istio-proxy" Jul 08 19:19:49.321293 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.321276 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:19:49.323920 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.323896 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ef94f1-openshift-default-dockercfg-snmvz\"" Jul 08 19:19:49.333284 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.332759 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f"] Jul 08 19:19:49.404154 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.404125 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1acd12b3-a32e-4827-bdca-f6918ec56586-workload-socket\") pod \"e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f\" (UID: \"1acd12b3-a32e-4827-bdca-f6918ec56586\") " pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:19:49.404154 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.404161 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1acd12b3-a32e-4827-bdca-f6918ec56586-istio-envoy\") pod \"e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f\" (UID: \"1acd12b3-a32e-4827-bdca-f6918ec56586\") " pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:19:49.404417 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.404189 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1acd12b3-a32e-4827-bdca-f6918ec56586-istiod-ca-cert\") pod \"e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f\" (UID: \"1acd12b3-a32e-4827-bdca-f6918ec56586\") " pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:19:49.404417 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.404304 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1acd12b3-a32e-4827-bdca-f6918ec56586-credential-socket\") pod \"e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f\" (UID: \"1acd12b3-a32e-4827-bdca-f6918ec56586\") " pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:19:49.404417 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.404363 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1acd12b3-a32e-4827-bdca-f6918ec56586-istio-data\") pod \"e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f\" (UID: \"1acd12b3-a32e-4827-bdca-f6918ec56586\") " pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:19:49.404417 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.404383 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1acd12b3-a32e-4827-bdca-f6918ec56586-istio-token\") pod \"e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f\" (UID: \"1acd12b3-a32e-4827-bdca-f6918ec56586\") " pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:19:49.404646 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.404436 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1acd12b3-a32e-4827-bdca-f6918ec56586-istio-podinfo\") pod \"e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f\" (UID: \"1acd12b3-a32e-4827-bdca-f6918ec56586\") " pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:19:49.404646 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.404507 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1acd12b3-a32e-4827-bdca-f6918ec56586-workload-certs\") pod \"e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f\" (UID: \"1acd12b3-a32e-4827-bdca-f6918ec56586\") " pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:19:49.404646 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.404531 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgs9x\" (UniqueName: \"kubernetes.io/projected/1acd12b3-a32e-4827-bdca-f6918ec56586-kube-api-access-mgs9x\") pod \"e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f\" (UID: \"1acd12b3-a32e-4827-bdca-f6918ec56586\") " pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:19:49.505723 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.505691 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1acd12b3-a32e-4827-bdca-f6918ec56586-credential-socket\") pod \"e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f\" (UID: \"1acd12b3-a32e-4827-bdca-f6918ec56586\") " pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:19:49.505893 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.505743 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1acd12b3-a32e-4827-bdca-f6918ec56586-istio-data\") pod \"e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f\" (UID: \"1acd12b3-a32e-4827-bdca-f6918ec56586\") " pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:19:49.505893 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.505774 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1acd12b3-a32e-4827-bdca-f6918ec56586-istio-token\") pod \"e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f\" (UID: \"1acd12b3-a32e-4827-bdca-f6918ec56586\") " pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:19:49.505893 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.505807 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1acd12b3-a32e-4827-bdca-f6918ec56586-istio-podinfo\") pod \"e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f\" (UID: \"1acd12b3-a32e-4827-bdca-f6918ec56586\") " pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:19:49.505893 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.505850 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1acd12b3-a32e-4827-bdca-f6918ec56586-workload-certs\") pod \"e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f\" (UID: \"1acd12b3-a32e-4827-bdca-f6918ec56586\") " pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:19:49.505893 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.505875 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgs9x\" (UniqueName: \"kubernetes.io/projected/1acd12b3-a32e-4827-bdca-f6918ec56586-kube-api-access-mgs9x\") pod \"e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f\" (UID: \"1acd12b3-a32e-4827-bdca-f6918ec56586\") " pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:19:49.506133 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.505914 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1acd12b3-a32e-4827-bdca-f6918ec56586-workload-socket\") pod \"e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f\" (UID: \"1acd12b3-a32e-4827-bdca-f6918ec56586\") " pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:19:49.506133 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.505950 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1acd12b3-a32e-4827-bdca-f6918ec56586-istio-envoy\") pod \"e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f\" (UID: \"1acd12b3-a32e-4827-bdca-f6918ec56586\") " pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:19:49.506133 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.505981 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1acd12b3-a32e-4827-bdca-f6918ec56586-istiod-ca-cert\") pod \"e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f\" (UID: \"1acd12b3-a32e-4827-bdca-f6918ec56586\") " pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:19:49.506133 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.506120 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1acd12b3-a32e-4827-bdca-f6918ec56586-credential-socket\") pod \"e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f\" (UID: \"1acd12b3-a32e-4827-bdca-f6918ec56586\") " pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:19:49.506373 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.506285 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1acd12b3-a32e-4827-bdca-f6918ec56586-workload-certs\") pod \"e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f\" (UID: \"1acd12b3-a32e-4827-bdca-f6918ec56586\") " pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:19:49.506373 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.506358 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1acd12b3-a32e-4827-bdca-f6918ec56586-istio-data\") pod \"e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f\" (UID: \"1acd12b3-a32e-4827-bdca-f6918ec56586\") " pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:19:49.506564 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.506546 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1acd12b3-a32e-4827-bdca-f6918ec56586-workload-socket\") pod \"e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f\" (UID: \"1acd12b3-a32e-4827-bdca-f6918ec56586\") " pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:19:49.506720 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.506703 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1acd12b3-a32e-4827-bdca-f6918ec56586-istiod-ca-cert\") pod \"e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f\" (UID: \"1acd12b3-a32e-4827-bdca-f6918ec56586\") " pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:19:49.508221 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.508203 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1acd12b3-a32e-4827-bdca-f6918ec56586-istio-envoy\") pod \"e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f\" (UID: \"1acd12b3-a32e-4827-bdca-f6918ec56586\") " pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:19:49.508307 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.508288 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1acd12b3-a32e-4827-bdca-f6918ec56586-istio-podinfo\") pod \"e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f\" (UID: \"1acd12b3-a32e-4827-bdca-f6918ec56586\") " pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:19:49.513564 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.513543 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1acd12b3-a32e-4827-bdca-f6918ec56586-istio-token\") pod \"e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f\" (UID: \"1acd12b3-a32e-4827-bdca-f6918ec56586\") " pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:19:49.513935 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.513916 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgs9x\" (UniqueName: \"kubernetes.io/projected/1acd12b3-a32e-4827-bdca-f6918ec56586-kube-api-access-mgs9x\") pod \"e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f\" (UID: \"1acd12b3-a32e-4827-bdca-f6918ec56586\") " pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:19:49.632881 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.632850 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:19:49.774099 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.774071 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f"] Jul 08 19:19:49.775082 ip-10-0-130-48 kubenswrapper[2570]: W0708 19:19:49.775059 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1acd12b3_a32e_4827_bdca_f6918ec56586.slice/crio-9d7376a0f4230a7cda8ef52a18da59f8af9bdd67920f1e9c4be9e11a74e58c3a WatchSource:0}: Error finding container 9d7376a0f4230a7cda8ef52a18da59f8af9bdd67920f1e9c4be9e11a74e58c3a: Status 404 returned error can't find the container with id 9d7376a0f4230a7cda8ef52a18da59f8af9bdd67920f1e9c4be9e11a74e58c3a Jul 08 19:19:49.777325 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.777286 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:19:49.777393 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.777355 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:19:49.777393 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:49.777386 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:19:50.124688 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:50.124654 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" event={"ID":"1acd12b3-a32e-4827-bdca-f6918ec56586","Type":"ContainerStarted","Data":"3937c4a73d0a7f81293707d54d96244bcbd293051224463d714acd73396de3be"} Jul 08 19:19:50.124891 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:50.124727 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" event={"ID":"1acd12b3-a32e-4827-bdca-f6918ec56586","Type":"ContainerStarted","Data":"9d7376a0f4230a7cda8ef52a18da59f8af9bdd67920f1e9c4be9e11a74e58c3a"} Jul 08 19:19:50.150587 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:50.150537 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" podStartSLOduration=1.150522301 podStartE2EDuration="1.150522301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:19:50.148176755 +0000 UTC m=+2050.995451422" watchObservedRunningTime="2026-07-08 19:19:50.150522301 +0000 UTC m=+2050.997796966" Jul 08 19:19:50.633078 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:50.633046 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:19:50.638470 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:50.638448 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:19:51.130055 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:51.130023 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:19:51.131064 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:51.131041 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:19:57.413865 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:57.413830 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v"] Jul 08 19:19:57.418060 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:57.418035 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:19:57.420797 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:57.420775 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c29bbf-openshift-default-dockercfg-rsf2g\"" Jul 08 19:19:57.431438 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:57.431411 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v"] Jul 08 19:19:57.480346 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:57.480302 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/756c32e6-e4d9-4707-858f-ce65dd86ac97-istiod-ca-cert\") pod \"e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v\" (UID: \"756c32e6-e4d9-4707-858f-ce65dd86ac97\") " pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:19:57.480346 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:57.480344 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/756c32e6-e4d9-4707-858f-ce65dd86ac97-istio-token\") pod \"e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v\" (UID: \"756c32e6-e4d9-4707-858f-ce65dd86ac97\") " pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:19:57.480578 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:57.480370 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/756c32e6-e4d9-4707-858f-ce65dd86ac97-istio-podinfo\") pod \"e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v\" (UID: \"756c32e6-e4d9-4707-858f-ce65dd86ac97\") " pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:19:57.480578 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:57.480505 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/756c32e6-e4d9-4707-858f-ce65dd86ac97-istio-envoy\") pod \"e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v\" (UID: \"756c32e6-e4d9-4707-858f-ce65dd86ac97\") " pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:19:57.480578 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:57.480534 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/756c32e6-e4d9-4707-858f-ce65dd86ac97-istio-data\") pod \"e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v\" (UID: \"756c32e6-e4d9-4707-858f-ce65dd86ac97\") " pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:19:57.480683 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:57.480577 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/756c32e6-e4d9-4707-858f-ce65dd86ac97-workload-certs\") pod \"e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v\" (UID: \"756c32e6-e4d9-4707-858f-ce65dd86ac97\") " pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:19:57.480683 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:57.480603 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/756c32e6-e4d9-4707-858f-ce65dd86ac97-credential-socket\") pod \"e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v\" (UID: \"756c32e6-e4d9-4707-858f-ce65dd86ac97\") " pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:19:57.480683 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:57.480623 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/756c32e6-e4d9-4707-858f-ce65dd86ac97-workload-socket\") pod \"e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v\" (UID: \"756c32e6-e4d9-4707-858f-ce65dd86ac97\") " pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:19:57.480683 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:57.480671 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7k8r\" (UniqueName: \"kubernetes.io/projected/756c32e6-e4d9-4707-858f-ce65dd86ac97-kube-api-access-k7k8r\") pod \"e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v\" (UID: \"756c32e6-e4d9-4707-858f-ce65dd86ac97\") " pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:19:57.582044 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:57.582010 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7k8r\" (UniqueName: \"kubernetes.io/projected/756c32e6-e4d9-4707-858f-ce65dd86ac97-kube-api-access-k7k8r\") pod \"e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v\" (UID: \"756c32e6-e4d9-4707-858f-ce65dd86ac97\") " pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:19:57.582242 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:57.582090 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/756c32e6-e4d9-4707-858f-ce65dd86ac97-istiod-ca-cert\") pod \"e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v\" (UID: \"756c32e6-e4d9-4707-858f-ce65dd86ac97\") " pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:19:57.582242 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:57.582162 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/756c32e6-e4d9-4707-858f-ce65dd86ac97-istio-token\") pod \"e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v\" (UID: \"756c32e6-e4d9-4707-858f-ce65dd86ac97\") " pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:19:57.582242 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:57.582220 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/756c32e6-e4d9-4707-858f-ce65dd86ac97-istio-podinfo\") pod \"e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v\" (UID: \"756c32e6-e4d9-4707-858f-ce65dd86ac97\") " pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:19:57.582473 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:57.582411 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/756c32e6-e4d9-4707-858f-ce65dd86ac97-istio-envoy\") pod \"e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v\" (UID: \"756c32e6-e4d9-4707-858f-ce65dd86ac97\") " pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:19:57.582473 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:57.582453 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/756c32e6-e4d9-4707-858f-ce65dd86ac97-istio-data\") pod \"e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v\" (UID: \"756c32e6-e4d9-4707-858f-ce65dd86ac97\") " pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:19:57.582582 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:57.582512 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/756c32e6-e4d9-4707-858f-ce65dd86ac97-workload-certs\") pod \"e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v\" (UID: \"756c32e6-e4d9-4707-858f-ce65dd86ac97\") " pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:19:57.582582 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:57.582548 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/756c32e6-e4d9-4707-858f-ce65dd86ac97-credential-socket\") pod \"e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v\" (UID: \"756c32e6-e4d9-4707-858f-ce65dd86ac97\") " pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:19:57.582687 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:57.582579 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/756c32e6-e4d9-4707-858f-ce65dd86ac97-workload-socket\") pod \"e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v\" (UID: \"756c32e6-e4d9-4707-858f-ce65dd86ac97\") " pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:19:57.582854 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:57.582827 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/756c32e6-e4d9-4707-858f-ce65dd86ac97-istio-data\") pod \"e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v\" (UID: \"756c32e6-e4d9-4707-858f-ce65dd86ac97\") " pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:19:57.583044 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:57.582886 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/756c32e6-e4d9-4707-858f-ce65dd86ac97-workload-certs\") pod \"e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v\" (UID: \"756c32e6-e4d9-4707-858f-ce65dd86ac97\") " pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:19:57.583044 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:57.582899 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/756c32e6-e4d9-4707-858f-ce65dd86ac97-credential-socket\") pod \"e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v\" (UID: \"756c32e6-e4d9-4707-858f-ce65dd86ac97\") " pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:19:57.583187 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:57.583048 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/756c32e6-e4d9-4707-858f-ce65dd86ac97-workload-socket\") pod \"e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v\" (UID: \"756c32e6-e4d9-4707-858f-ce65dd86ac97\") " pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:19:57.583187 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:57.582919 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/756c32e6-e4d9-4707-858f-ce65dd86ac97-istiod-ca-cert\") pod \"e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v\" (UID: \"756c32e6-e4d9-4707-858f-ce65dd86ac97\") " pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:19:57.584783 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:57.584756 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/756c32e6-e4d9-4707-858f-ce65dd86ac97-istio-envoy\") pod \"e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v\" (UID: \"756c32e6-e4d9-4707-858f-ce65dd86ac97\") " pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:19:57.584783 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:57.584775 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/756c32e6-e4d9-4707-858f-ce65dd86ac97-istio-podinfo\") pod \"e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v\" (UID: \"756c32e6-e4d9-4707-858f-ce65dd86ac97\") " pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:19:57.599140 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:57.599110 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7k8r\" (UniqueName: \"kubernetes.io/projected/756c32e6-e4d9-4707-858f-ce65dd86ac97-kube-api-access-k7k8r\") pod \"e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v\" (UID: \"756c32e6-e4d9-4707-858f-ce65dd86ac97\") " pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:19:57.599356 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:57.599341 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/756c32e6-e4d9-4707-858f-ce65dd86ac97-istio-token\") pod \"e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v\" (UID: \"756c32e6-e4d9-4707-858f-ce65dd86ac97\") " pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:19:57.730389 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:57.730303 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:19:57.875411 ip-10-0-130-48 kubenswrapper[2570]: W0708 19:19:57.875383 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod756c32e6_e4d9_4707_858f_ce65dd86ac97.slice/crio-adae4ff3b719e7424225b1981768ca5150df585899c1089cfab7066e7480c593 WatchSource:0}: Error finding container adae4ff3b719e7424225b1981768ca5150df585899c1089cfab7066e7480c593: Status 404 returned error can't find the container with id adae4ff3b719e7424225b1981768ca5150df585899c1089cfab7066e7480c593 Jul 08 19:19:57.875545 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:57.875524 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v"] Jul 08 19:19:57.877494 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:57.877464 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:19:57.877574 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:57.877524 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:19:57.877574 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:57.877552 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:19:58.179917 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:58.179883 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" event={"ID":"756c32e6-e4d9-4707-858f-ce65dd86ac97","Type":"ContainerStarted","Data":"32bb4c9e2f5a23a3114dc6028d794d62591a428dee311d97aae3cf07ba2eb63e"} Jul 08 19:19:58.180072 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:58.179941 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" event={"ID":"756c32e6-e4d9-4707-858f-ce65dd86ac97","Type":"ContainerStarted","Data":"adae4ff3b719e7424225b1981768ca5150df585899c1089cfab7066e7480c593"} Jul 08 19:19:58.202663 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:58.202610 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" podStartSLOduration=1.202591227 podStartE2EDuration="1.202591227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:19:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:19:58.200294042 +0000 UTC m=+2059.047568711" watchObservedRunningTime="2026-07-08 19:19:58.202591227 +0000 UTC m=+2059.049865894" Jul 08 19:19:58.730953 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:58.730915 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:19:58.736514 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:58.736487 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:19:59.185274 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:59.185218 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:19:59.186499 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:59.186475 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:19:59.998866 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:19:59.998834 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:20:07.171454 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:07.171409 2570 patch_prober.go:28] interesting pod/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 08 19:20:07.171866 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:07.171495 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" podUID="1acd12b3-a32e-4827-bdca-f6918ec56586" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 08 19:20:09.246062 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:09.246027 2570 generic.go:356] "Generic (PLEG): container finished" podID="1acd12b3-a32e-4827-bdca-f6918ec56586" containerID="3937c4a73d0a7f81293707d54d96244bcbd293051224463d714acd73396de3be" exitCode=0 Jul 08 19:20:09.246558 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:09.246097 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" event={"ID":"1acd12b3-a32e-4827-bdca-f6918ec56586","Type":"ContainerDied","Data":"3937c4a73d0a7f81293707d54d96244bcbd293051224463d714acd73396de3be"} Jul 08 19:20:09.246657 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:09.246571 2570 scope.go:117] "RemoveContainer" containerID="3937c4a73d0a7f81293707d54d96244bcbd293051224463d714acd73396de3be" Jul 08 19:20:09.247696 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:09.247664 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:20:09.247777 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:09.247758 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:20:09.247819 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:09.247805 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:20:10.261775 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:10.261733 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" event={"ID":"1acd12b3-a32e-4827-bdca-f6918ec56586","Type":"ContainerStarted","Data":"6573611498a869624d4f3b22ac1ec2162218c447ed1446c52ccada9cbae86d23"} Jul 08 19:20:10.633421 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:10.633390 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:20:10.633879 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:10.633852 2570 patch_prober.go:28] interesting pod/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Jul 08 19:20:10.634108 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:10.634065 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" podUID="1acd12b3-a32e-4827-bdca-f6918ec56586" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Jul 08 19:20:11.634243 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:11.633792 2570 patch_prober.go:28] interesting pod/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Jul 08 19:20:11.634243 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:11.633853 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" podUID="1acd12b3-a32e-4827-bdca-f6918ec56586" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Jul 08 19:20:12.633389 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:12.633350 2570 patch_prober.go:28] interesting pod/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Jul 08 19:20:12.633685 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:12.633654 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" podUID="1acd12b3-a32e-4827-bdca-f6918ec56586" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Jul 08 19:20:13.634903 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:13.634853 2570 patch_prober.go:28] interesting pod/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Jul 08 19:20:13.635529 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:13.635476 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" podUID="1acd12b3-a32e-4827-bdca-f6918ec56586" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Jul 08 19:20:14.639046 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:14.639015 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:20:14.639546 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:14.639365 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:20:14.640156 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:14.640138 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:20:17.691380 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:17.691351 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:20:18.321996 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:18.321966 2570 generic.go:356] "Generic (PLEG): container finished" podID="756c32e6-e4d9-4707-858f-ce65dd86ac97" containerID="32bb4c9e2f5a23a3114dc6028d794d62591a428dee311d97aae3cf07ba2eb63e" exitCode=0 Jul 08 19:20:18.322176 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:18.322047 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" event={"ID":"756c32e6-e4d9-4707-858f-ce65dd86ac97","Type":"ContainerDied","Data":"32bb4c9e2f5a23a3114dc6028d794d62591a428dee311d97aae3cf07ba2eb63e"} Jul 08 19:20:18.322510 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:18.322496 2570 scope.go:117] "RemoveContainer" containerID="32bb4c9e2f5a23a3114dc6028d794d62591a428dee311d97aae3cf07ba2eb63e" Jul 08 19:20:18.323522 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:18.323497 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:20:18.323581 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:18.323561 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:20:18.323620 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:18.323593 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:20:19.329643 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:19.329575 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" event={"ID":"756c32e6-e4d9-4707-858f-ce65dd86ac97","Type":"ContainerStarted","Data":"bde50cb2c05a023fe85973064747d963440471a2153554be6d6c58d493f0a326"} Jul 08 19:20:19.731894 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:19.731477 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:20:19.731894 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:19.731599 2570 patch_prober.go:28] interesting pod/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Jul 08 19:20:19.731894 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:19.731644 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" podUID="756c32e6-e4d9-4707-858f-ce65dd86ac97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Jul 08 19:20:20.378296 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:20.378248 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f"] Jul 08 19:20:20.378789 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:20.378471 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" podUID="1acd12b3-a32e-4827-bdca-f6918ec56586" containerName="istio-proxy" containerID="cri-o://6573611498a869624d4f3b22ac1ec2162218c447ed1446c52ccada9cbae86d23" gracePeriod=30 Jul 08 19:20:20.731453 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:20.731363 2570 patch_prober.go:28] interesting pod/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Jul 08 19:20:20.731747 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:20.731716 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" podUID="756c32e6-e4d9-4707-858f-ce65dd86ac97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Jul 08 19:20:21.731319 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:21.731282 2570 patch_prober.go:28] interesting pod/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Jul 08 19:20:21.732433 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:21.731993 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" podUID="756c32e6-e4d9-4707-858f-ce65dd86ac97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Jul 08 19:20:22.731472 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:22.731433 2570 patch_prober.go:28] interesting pod/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Jul 08 19:20:22.732051 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:22.732018 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" podUID="756c32e6-e4d9-4707-858f-ce65dd86ac97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Jul 08 19:20:23.734382 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:23.734352 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:20:23.734869 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:23.734855 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:20:23.735734 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:23.735714 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:20:25.633003 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:25.632978 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:20:25.678742 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:25.678670 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1acd12b3-a32e-4827-bdca-f6918ec56586-istio-podinfo\") pod \"1acd12b3-a32e-4827-bdca-f6918ec56586\" (UID: \"1acd12b3-a32e-4827-bdca-f6918ec56586\") " Jul 08 19:20:25.678742 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:25.678717 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1acd12b3-a32e-4827-bdca-f6918ec56586-istio-envoy\") pod \"1acd12b3-a32e-4827-bdca-f6918ec56586\" (UID: \"1acd12b3-a32e-4827-bdca-f6918ec56586\") " Jul 08 19:20:25.678974 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:25.678751 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1acd12b3-a32e-4827-bdca-f6918ec56586-workload-socket\") pod \"1acd12b3-a32e-4827-bdca-f6918ec56586\" (UID: \"1acd12b3-a32e-4827-bdca-f6918ec56586\") " Jul 08 19:20:25.678974 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:25.678821 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgs9x\" (UniqueName: \"kubernetes.io/projected/1acd12b3-a32e-4827-bdca-f6918ec56586-kube-api-access-mgs9x\") pod \"1acd12b3-a32e-4827-bdca-f6918ec56586\" (UID: \"1acd12b3-a32e-4827-bdca-f6918ec56586\") " Jul 08 19:20:25.679128 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:25.679000 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1acd12b3-a32e-4827-bdca-f6918ec56586-credential-socket\") pod \"1acd12b3-a32e-4827-bdca-f6918ec56586\" (UID: \"1acd12b3-a32e-4827-bdca-f6918ec56586\") " Jul 08 19:20:25.679128 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:25.679030 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1acd12b3-a32e-4827-bdca-f6918ec56586-istio-token\") pod \"1acd12b3-a32e-4827-bdca-f6918ec56586\" (UID: \"1acd12b3-a32e-4827-bdca-f6918ec56586\") " Jul 08 19:20:25.679128 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:25.679071 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1acd12b3-a32e-4827-bdca-f6918ec56586-workload-certs\") pod \"1acd12b3-a32e-4827-bdca-f6918ec56586\" (UID: \"1acd12b3-a32e-4827-bdca-f6918ec56586\") " Jul 08 19:20:25.679128 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:25.679105 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1acd12b3-a32e-4827-bdca-f6918ec56586-istiod-ca-cert\") pod \"1acd12b3-a32e-4827-bdca-f6918ec56586\" (UID: \"1acd12b3-a32e-4827-bdca-f6918ec56586\") " Jul 08 19:20:25.679369 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:25.679195 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1acd12b3-a32e-4827-bdca-f6918ec56586-istio-data\") pod \"1acd12b3-a32e-4827-bdca-f6918ec56586\" (UID: \"1acd12b3-a32e-4827-bdca-f6918ec56586\") " Jul 08 19:20:25.679369 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:25.679286 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1acd12b3-a32e-4827-bdca-f6918ec56586-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1acd12b3-a32e-4827-bdca-f6918ec56586" (UID: "1acd12b3-a32e-4827-bdca-f6918ec56586"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:20:25.679489 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:25.679428 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1acd12b3-a32e-4827-bdca-f6918ec56586-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1acd12b3-a32e-4827-bdca-f6918ec56586" (UID: "1acd12b3-a32e-4827-bdca-f6918ec56586"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:20:25.679959 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:25.679554 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1acd12b3-a32e-4827-bdca-f6918ec56586-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1acd12b3-a32e-4827-bdca-f6918ec56586" (UID: "1acd12b3-a32e-4827-bdca-f6918ec56586"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:20:25.679959 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:25.679627 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1acd12b3-a32e-4827-bdca-f6918ec56586-workload-socket\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:20:25.679959 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:25.679649 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1acd12b3-a32e-4827-bdca-f6918ec56586-credential-socket\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:20:25.679959 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:25.679904 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1acd12b3-a32e-4827-bdca-f6918ec56586-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1acd12b3-a32e-4827-bdca-f6918ec56586" (UID: "1acd12b3-a32e-4827-bdca-f6918ec56586"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:20:25.680348 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:25.680320 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1acd12b3-a32e-4827-bdca-f6918ec56586-istio-data" (OuterVolumeSpecName: "istio-data") pod "1acd12b3-a32e-4827-bdca-f6918ec56586" (UID: "1acd12b3-a32e-4827-bdca-f6918ec56586"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:20:25.681233 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:25.681206 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1acd12b3-a32e-4827-bdca-f6918ec56586-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1acd12b3-a32e-4827-bdca-f6918ec56586" (UID: "1acd12b3-a32e-4827-bdca-f6918ec56586"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:20:25.681406 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:25.681275 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1acd12b3-a32e-4827-bdca-f6918ec56586-kube-api-access-mgs9x" (OuterVolumeSpecName: "kube-api-access-mgs9x") pod "1acd12b3-a32e-4827-bdca-f6918ec56586" (UID: "1acd12b3-a32e-4827-bdca-f6918ec56586"). InnerVolumeSpecName "kube-api-access-mgs9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:20:25.681569 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:25.681547 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1acd12b3-a32e-4827-bdca-f6918ec56586-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1acd12b3-a32e-4827-bdca-f6918ec56586" (UID: "1acd12b3-a32e-4827-bdca-f6918ec56586"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 19:20:25.681729 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:25.681706 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1acd12b3-a32e-4827-bdca-f6918ec56586-istio-token" (OuterVolumeSpecName: "istio-token") pod "1acd12b3-a32e-4827-bdca-f6918ec56586" (UID: "1acd12b3-a32e-4827-bdca-f6918ec56586"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:20:25.780517 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:25.780481 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1acd12b3-a32e-4827-bdca-f6918ec56586-istio-data\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:20:25.780517 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:25.780512 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1acd12b3-a32e-4827-bdca-f6918ec56586-istio-podinfo\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:20:25.780719 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:25.780526 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1acd12b3-a32e-4827-bdca-f6918ec56586-istio-envoy\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:20:25.780719 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:25.780538 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgs9x\" (UniqueName: \"kubernetes.io/projected/1acd12b3-a32e-4827-bdca-f6918ec56586-kube-api-access-mgs9x\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:20:25.780719 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:25.780552 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1acd12b3-a32e-4827-bdca-f6918ec56586-istio-token\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:20:25.780719 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:25.780564 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1acd12b3-a32e-4827-bdca-f6918ec56586-workload-certs\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:20:25.780719 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:25.780578 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1acd12b3-a32e-4827-bdca-f6918ec56586-istiod-ca-cert\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:20:26.383842 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:26.383803 2570 generic.go:356] "Generic (PLEG): container finished" podID="1acd12b3-a32e-4827-bdca-f6918ec56586" containerID="6573611498a869624d4f3b22ac1ec2162218c447ed1446c52ccada9cbae86d23" exitCode=0 Jul 08 19:20:26.383842 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:26.383844 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" Jul 08 19:20:26.384102 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:26.383885 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" event={"ID":"1acd12b3-a32e-4827-bdca-f6918ec56586","Type":"ContainerDied","Data":"6573611498a869624d4f3b22ac1ec2162218c447ed1446c52ccada9cbae86d23"} Jul 08 19:20:26.384102 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:26.383945 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f" event={"ID":"1acd12b3-a32e-4827-bdca-f6918ec56586","Type":"ContainerDied","Data":"9d7376a0f4230a7cda8ef52a18da59f8af9bdd67920f1e9c4be9e11a74e58c3a"} Jul 08 19:20:26.384102 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:26.383970 2570 scope.go:117] "RemoveContainer" containerID="6573611498a869624d4f3b22ac1ec2162218c447ed1446c52ccada9cbae86d23" Jul 08 19:20:26.393485 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:26.393460 2570 scope.go:117] "RemoveContainer" containerID="3937c4a73d0a7f81293707d54d96244bcbd293051224463d714acd73396de3be" Jul 08 19:20:26.403994 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:26.403772 2570 scope.go:117] "RemoveContainer" containerID="6573611498a869624d4f3b22ac1ec2162218c447ed1446c52ccada9cbae86d23" Jul 08 19:20:26.405296 ip-10-0-130-48 kubenswrapper[2570]: E0708 19:20:26.405249 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6573611498a869624d4f3b22ac1ec2162218c447ed1446c52ccada9cbae86d23\": container with ID starting with 6573611498a869624d4f3b22ac1ec2162218c447ed1446c52ccada9cbae86d23 not found: ID does not exist" containerID="6573611498a869624d4f3b22ac1ec2162218c447ed1446c52ccada9cbae86d23" Jul 08 19:20:26.405412 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:26.405307 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6573611498a869624d4f3b22ac1ec2162218c447ed1446c52ccada9cbae86d23"} err="failed to get container status \"6573611498a869624d4f3b22ac1ec2162218c447ed1446c52ccada9cbae86d23\": rpc error: code = NotFound desc = could not find container \"6573611498a869624d4f3b22ac1ec2162218c447ed1446c52ccada9cbae86d23\": container with ID starting with 6573611498a869624d4f3b22ac1ec2162218c447ed1446c52ccada9cbae86d23 not found: ID does not exist" Jul 08 19:20:26.405412 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:26.405332 2570 scope.go:117] "RemoveContainer" containerID="3937c4a73d0a7f81293707d54d96244bcbd293051224463d714acd73396de3be" Jul 08 19:20:26.405660 ip-10-0-130-48 kubenswrapper[2570]: E0708 19:20:26.405631 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3937c4a73d0a7f81293707d54d96244bcbd293051224463d714acd73396de3be\": container with ID starting with 3937c4a73d0a7f81293707d54d96244bcbd293051224463d714acd73396de3be not found: ID does not exist" containerID="3937c4a73d0a7f81293707d54d96244bcbd293051224463d714acd73396de3be" Jul 08 19:20:26.405716 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:26.405669 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3937c4a73d0a7f81293707d54d96244bcbd293051224463d714acd73396de3be"} err="failed to get container status \"3937c4a73d0a7f81293707d54d96244bcbd293051224463d714acd73396de3be\": rpc error: code = NotFound desc = could not find container \"3937c4a73d0a7f81293707d54d96244bcbd293051224463d714acd73396de3be\": container with ID starting with 3937c4a73d0a7f81293707d54d96244bcbd293051224463d714acd73396de3be not found: ID does not exist" Jul 08 19:20:26.405767 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:26.405758 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f"] Jul 08 19:20:26.408135 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:26.408116 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ef94f1-openshift-default-7ffc994fdf-t4k2f"] Jul 08 19:20:27.694275 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:27.691649 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1acd12b3-a32e-4827-bdca-f6918ec56586" path="/var/lib/kubelet/pods/1acd12b3-a32e-4827-bdca-f6918ec56586/volumes" Jul 08 19:20:27.748207 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:27.748171 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v"] Jul 08 19:20:27.748422 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:27.748398 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" podUID="756c32e6-e4d9-4707-858f-ce65dd86ac97" containerName="istio-proxy" containerID="cri-o://bde50cb2c05a023fe85973064747d963440471a2153554be6d6c58d493f0a326" gracePeriod=30 Jul 08 19:20:32.128432 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:32.128399 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:20:33.016987 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.016919 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:20:33.148126 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.148089 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/756c32e6-e4d9-4707-858f-ce65dd86ac97-istio-token\") pod \"756c32e6-e4d9-4707-858f-ce65dd86ac97\" (UID: \"756c32e6-e4d9-4707-858f-ce65dd86ac97\") " Jul 08 19:20:33.148653 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.148165 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/756c32e6-e4d9-4707-858f-ce65dd86ac97-istiod-ca-cert\") pod \"756c32e6-e4d9-4707-858f-ce65dd86ac97\" (UID: \"756c32e6-e4d9-4707-858f-ce65dd86ac97\") " Jul 08 19:20:33.148653 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.148188 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/756c32e6-e4d9-4707-858f-ce65dd86ac97-istio-podinfo\") pod \"756c32e6-e4d9-4707-858f-ce65dd86ac97\" (UID: \"756c32e6-e4d9-4707-858f-ce65dd86ac97\") " Jul 08 19:20:33.148653 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.148210 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/756c32e6-e4d9-4707-858f-ce65dd86ac97-istio-data\") pod \"756c32e6-e4d9-4707-858f-ce65dd86ac97\" (UID: \"756c32e6-e4d9-4707-858f-ce65dd86ac97\") " Jul 08 19:20:33.148653 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.148226 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/756c32e6-e4d9-4707-858f-ce65dd86ac97-credential-socket\") pod \"756c32e6-e4d9-4707-858f-ce65dd86ac97\" (UID: \"756c32e6-e4d9-4707-858f-ce65dd86ac97\") " Jul 08 19:20:33.148653 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.148280 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/756c32e6-e4d9-4707-858f-ce65dd86ac97-workload-certs\") pod \"756c32e6-e4d9-4707-858f-ce65dd86ac97\" (UID: \"756c32e6-e4d9-4707-858f-ce65dd86ac97\") " Jul 08 19:20:33.148653 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.148323 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/756c32e6-e4d9-4707-858f-ce65dd86ac97-istio-envoy\") pod \"756c32e6-e4d9-4707-858f-ce65dd86ac97\" (UID: \"756c32e6-e4d9-4707-858f-ce65dd86ac97\") " Jul 08 19:20:33.148653 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.148348 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7k8r\" (UniqueName: \"kubernetes.io/projected/756c32e6-e4d9-4707-858f-ce65dd86ac97-kube-api-access-k7k8r\") pod \"756c32e6-e4d9-4707-858f-ce65dd86ac97\" (UID: \"756c32e6-e4d9-4707-858f-ce65dd86ac97\") " Jul 08 19:20:33.148653 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.148415 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/756c32e6-e4d9-4707-858f-ce65dd86ac97-workload-socket\") pod \"756c32e6-e4d9-4707-858f-ce65dd86ac97\" (UID: \"756c32e6-e4d9-4707-858f-ce65dd86ac97\") " Jul 08 19:20:33.148653 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.148556 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/756c32e6-e4d9-4707-858f-ce65dd86ac97-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "756c32e6-e4d9-4707-858f-ce65dd86ac97" (UID: "756c32e6-e4d9-4707-858f-ce65dd86ac97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:20:33.148653 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.148605 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/756c32e6-e4d9-4707-858f-ce65dd86ac97-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "756c32e6-e4d9-4707-858f-ce65dd86ac97" (UID: "756c32e6-e4d9-4707-858f-ce65dd86ac97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:20:33.148653 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.148654 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/756c32e6-e4d9-4707-858f-ce65dd86ac97-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "756c32e6-e4d9-4707-858f-ce65dd86ac97" (UID: "756c32e6-e4d9-4707-858f-ce65dd86ac97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:20:33.149669 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.148789 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/756c32e6-e4d9-4707-858f-ce65dd86ac97-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "756c32e6-e4d9-4707-858f-ce65dd86ac97" (UID: "756c32e6-e4d9-4707-858f-ce65dd86ac97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:20:33.149669 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.148936 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/756c32e6-e4d9-4707-858f-ce65dd86ac97-credential-socket\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:20:33.149669 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.148959 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/756c32e6-e4d9-4707-858f-ce65dd86ac97-workload-certs\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:20:33.149669 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.148976 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/756c32e6-e4d9-4707-858f-ce65dd86ac97-workload-socket\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:20:33.149669 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.148991 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/756c32e6-e4d9-4707-858f-ce65dd86ac97-istiod-ca-cert\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:20:33.149669 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.149458 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/756c32e6-e4d9-4707-858f-ce65dd86ac97-istio-data" (OuterVolumeSpecName: "istio-data") pod "756c32e6-e4d9-4707-858f-ce65dd86ac97" (UID: "756c32e6-e4d9-4707-858f-ce65dd86ac97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:20:33.150700 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.150665 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/756c32e6-e4d9-4707-858f-ce65dd86ac97-kube-api-access-k7k8r" (OuterVolumeSpecName: "kube-api-access-k7k8r") pod "756c32e6-e4d9-4707-858f-ce65dd86ac97" (UID: "756c32e6-e4d9-4707-858f-ce65dd86ac97"). InnerVolumeSpecName "kube-api-access-k7k8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:20:33.150804 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.150700 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/756c32e6-e4d9-4707-858f-ce65dd86ac97-istio-token" (OuterVolumeSpecName: "istio-token") pod "756c32e6-e4d9-4707-858f-ce65dd86ac97" (UID: "756c32e6-e4d9-4707-858f-ce65dd86ac97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:20:33.150877 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.150859 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/756c32e6-e4d9-4707-858f-ce65dd86ac97-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "756c32e6-e4d9-4707-858f-ce65dd86ac97" (UID: "756c32e6-e4d9-4707-858f-ce65dd86ac97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:20:33.150920 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.150861 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/756c32e6-e4d9-4707-858f-ce65dd86ac97-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "756c32e6-e4d9-4707-858f-ce65dd86ac97" (UID: "756c32e6-e4d9-4707-858f-ce65dd86ac97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 19:20:33.250024 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.249986 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/756c32e6-e4d9-4707-858f-ce65dd86ac97-istio-podinfo\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:20:33.250024 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.250016 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/756c32e6-e4d9-4707-858f-ce65dd86ac97-istio-data\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:20:33.250024 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.250024 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/756c32e6-e4d9-4707-858f-ce65dd86ac97-istio-envoy\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:20:33.250024 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.250033 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7k8r\" (UniqueName: \"kubernetes.io/projected/756c32e6-e4d9-4707-858f-ce65dd86ac97-kube-api-access-k7k8r\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:20:33.250348 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.250044 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/756c32e6-e4d9-4707-858f-ce65dd86ac97-istio-token\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:20:33.426907 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.426868 2570 generic.go:356] "Generic (PLEG): container finished" podID="756c32e6-e4d9-4707-858f-ce65dd86ac97" containerID="bde50cb2c05a023fe85973064747d963440471a2153554be6d6c58d493f0a326" exitCode=0 Jul 08 19:20:33.427099 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.426924 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" Jul 08 19:20:33.427099 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.426946 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" event={"ID":"756c32e6-e4d9-4707-858f-ce65dd86ac97","Type":"ContainerDied","Data":"bde50cb2c05a023fe85973064747d963440471a2153554be6d6c58d493f0a326"} Jul 08 19:20:33.427099 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.427002 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v" event={"ID":"756c32e6-e4d9-4707-858f-ce65dd86ac97","Type":"ContainerDied","Data":"adae4ff3b719e7424225b1981768ca5150df585899c1089cfab7066e7480c593"} Jul 08 19:20:33.427099 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.427024 2570 scope.go:117] "RemoveContainer" containerID="bde50cb2c05a023fe85973064747d963440471a2153554be6d6c58d493f0a326" Jul 08 19:20:33.438075 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.438057 2570 scope.go:117] "RemoveContainer" containerID="32bb4c9e2f5a23a3114dc6028d794d62591a428dee311d97aae3cf07ba2eb63e" Jul 08 19:20:33.446214 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.446192 2570 scope.go:117] "RemoveContainer" containerID="bde50cb2c05a023fe85973064747d963440471a2153554be6d6c58d493f0a326" Jul 08 19:20:33.446473 ip-10-0-130-48 kubenswrapper[2570]: E0708 19:20:33.446453 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bde50cb2c05a023fe85973064747d963440471a2153554be6d6c58d493f0a326\": container with ID starting with bde50cb2c05a023fe85973064747d963440471a2153554be6d6c58d493f0a326 not found: ID does not exist" containerID="bde50cb2c05a023fe85973064747d963440471a2153554be6d6c58d493f0a326" Jul 08 19:20:33.446524 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.446482 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde50cb2c05a023fe85973064747d963440471a2153554be6d6c58d493f0a326"} err="failed to get container status \"bde50cb2c05a023fe85973064747d963440471a2153554be6d6c58d493f0a326\": rpc error: code = NotFound desc = could not find container \"bde50cb2c05a023fe85973064747d963440471a2153554be6d6c58d493f0a326\": container with ID starting with bde50cb2c05a023fe85973064747d963440471a2153554be6d6c58d493f0a326 not found: ID does not exist" Jul 08 19:20:33.446524 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.446502 2570 scope.go:117] "RemoveContainer" containerID="32bb4c9e2f5a23a3114dc6028d794d62591a428dee311d97aae3cf07ba2eb63e" Jul 08 19:20:33.446740 ip-10-0-130-48 kubenswrapper[2570]: E0708 19:20:33.446723 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32bb4c9e2f5a23a3114dc6028d794d62591a428dee311d97aae3cf07ba2eb63e\": container with ID starting with 32bb4c9e2f5a23a3114dc6028d794d62591a428dee311d97aae3cf07ba2eb63e not found: ID does not exist" containerID="32bb4c9e2f5a23a3114dc6028d794d62591a428dee311d97aae3cf07ba2eb63e" Jul 08 19:20:33.446783 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.446745 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32bb4c9e2f5a23a3114dc6028d794d62591a428dee311d97aae3cf07ba2eb63e"} err="failed to get container status \"32bb4c9e2f5a23a3114dc6028d794d62591a428dee311d97aae3cf07ba2eb63e\": rpc error: code = NotFound desc = could not find container \"32bb4c9e2f5a23a3114dc6028d794d62591a428dee311d97aae3cf07ba2eb63e\": container with ID starting with 32bb4c9e2f5a23a3114dc6028d794d62591a428dee311d97aae3cf07ba2eb63e not found: ID does not exist" Jul 08 19:20:33.454908 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.454879 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v"] Jul 08 19:20:33.462175 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.461563 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c29bbf-openshift-default-6997d9d89-9cn7v"] Jul 08 19:20:33.690844 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:33.690767 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="756c32e6-e4d9-4707-858f-ce65dd86ac97" path="/var/lib/kubelet/pods/756c32e6-e4d9-4707-858f-ce65dd86ac97/volumes" Jul 08 19:20:36.968477 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:36.968449 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb"] Jul 08 19:20:36.968876 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:36.968862 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1acd12b3-a32e-4827-bdca-f6918ec56586" containerName="istio-proxy" Jul 08 19:20:36.968876 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:36.968877 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1acd12b3-a32e-4827-bdca-f6918ec56586" containerName="istio-proxy" Jul 08 19:20:36.968954 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:36.968896 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="756c32e6-e4d9-4707-858f-ce65dd86ac97" containerName="istio-proxy" Jul 08 19:20:36.968954 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:36.968902 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="756c32e6-e4d9-4707-858f-ce65dd86ac97" containerName="istio-proxy" Jul 08 19:20:36.968954 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:36.968909 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="756c32e6-e4d9-4707-858f-ce65dd86ac97" containerName="istio-proxy" Jul 08 19:20:36.968954 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:36.968915 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="756c32e6-e4d9-4707-858f-ce65dd86ac97" containerName="istio-proxy" Jul 08 19:20:36.968954 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:36.968923 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1acd12b3-a32e-4827-bdca-f6918ec56586" containerName="istio-proxy" Jul 08 19:20:36.968954 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:36.968928 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1acd12b3-a32e-4827-bdca-f6918ec56586" containerName="istio-proxy" Jul 08 19:20:36.969135 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:36.969002 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="756c32e6-e4d9-4707-858f-ce65dd86ac97" containerName="istio-proxy" Jul 08 19:20:36.969135 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:36.969012 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="1acd12b3-a32e-4827-bdca-f6918ec56586" containerName="istio-proxy" Jul 08 19:20:36.969135 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:36.969019 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="756c32e6-e4d9-4707-858f-ce65dd86ac97" containerName="istio-proxy" Jul 08 19:20:36.973870 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:36.973848 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:20:36.976509 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:36.976489 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-b28b28-openshift-default-dockercfg-lt9qr\"" Jul 08 19:20:36.983656 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:36.983633 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb"] Jul 08 19:20:37.086272 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:37.086226 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-istio-podinfo\") pod \"e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb\" (UID: \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\") " pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:20:37.086467 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:37.086297 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-istio-envoy\") pod \"e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb\" (UID: \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\") " pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:20:37.086467 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:37.086318 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhmh7\" (UniqueName: \"kubernetes.io/projected/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-kube-api-access-xhmh7\") pod \"e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb\" (UID: \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\") " pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:20:37.086467 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:37.086344 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-workload-certs\") pod \"e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb\" (UID: \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\") " pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:20:37.086467 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:37.086419 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-credential-socket\") pod \"e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb\" (UID: \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\") " pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:20:37.086467 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:37.086439 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-istio-data\") pod \"e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb\" (UID: \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\") " pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:20:37.086467 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:37.086461 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-istio-token\") pod \"e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb\" (UID: \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\") " pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:20:37.086790 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:37.086491 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-workload-socket\") pod \"e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb\" (UID: \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\") " pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:20:37.086790 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:37.086571 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-istiod-ca-cert\") pod \"e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb\" (UID: \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\") " pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:20:37.187587 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:37.187543 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-istiod-ca-cert\") pod \"e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb\" (UID: \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\") " pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:20:37.187758 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:37.187627 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-istio-podinfo\") pod \"e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb\" (UID: \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\") " pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:20:37.187758 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:37.187662 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-istio-envoy\") pod \"e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb\" (UID: \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\") " pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:20:37.187758 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:37.187684 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhmh7\" (UniqueName: \"kubernetes.io/projected/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-kube-api-access-xhmh7\") pod \"e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb\" (UID: \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\") " pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:20:37.187758 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:37.187714 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-workload-certs\") pod \"e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb\" (UID: \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\") " pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:20:37.187758 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:37.187756 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-credential-socket\") pod \"e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb\" (UID: \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\") " pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:20:37.188017 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:37.187781 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-istio-data\") pod \"e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb\" (UID: \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\") " pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:20:37.188017 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:37.187834 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-istio-token\") pod \"e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb\" (UID: \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\") " pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:20:37.188017 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:37.187928 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-workload-socket\") pod \"e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb\" (UID: \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\") " pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:20:37.188200 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:37.188161 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-workload-certs\") pod \"e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb\" (UID: \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\") " pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:20:37.188200 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:37.188187 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-credential-socket\") pod \"e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb\" (UID: \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\") " pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:20:37.188328 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:37.188239 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-istio-data\") pod \"e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb\" (UID: \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\") " pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:20:37.188462 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:37.188438 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-istiod-ca-cert\") pod \"e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb\" (UID: \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\") " pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:20:37.188614 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:37.188504 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-workload-socket\") pod \"e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb\" (UID: \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\") " pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:20:37.189980 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:37.189959 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-istio-envoy\") pod \"e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb\" (UID: \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\") " pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:20:37.190305 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:37.190286 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-istio-podinfo\") pod \"e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb\" (UID: \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\") " pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:20:37.200071 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:37.200045 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhmh7\" (UniqueName: \"kubernetes.io/projected/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-kube-api-access-xhmh7\") pod \"e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb\" (UID: \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\") " pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:20:37.201239 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:37.201219 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-istio-token\") pod \"e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb\" (UID: \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\") " pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:20:37.286122 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:37.286036 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:20:37.427673 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:37.427645 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb"] Jul 08 19:20:37.429542 ip-10-0-130-48 kubenswrapper[2570]: W0708 19:20:37.429509 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35c8b38e_f351_4a13_ba10_5cfbdd7f0acd.slice/crio-51caba670fc4d5f48222b87814ff68e4609c6d17542eee05773c05f6a1671e3a WatchSource:0}: Error finding container 51caba670fc4d5f48222b87814ff68e4609c6d17542eee05773c05f6a1671e3a: Status 404 returned error can't find the container with id 51caba670fc4d5f48222b87814ff68e4609c6d17542eee05773c05f6a1671e3a Jul 08 19:20:37.432212 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:37.432168 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:20:37.432358 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:37.432306 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:20:37.432406 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:37.432361 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:20:37.454454 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:37.454427 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" event={"ID":"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd","Type":"ContainerStarted","Data":"51caba670fc4d5f48222b87814ff68e4609c6d17542eee05773c05f6a1671e3a"} Jul 08 19:20:38.461660 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:38.461626 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" event={"ID":"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd","Type":"ContainerStarted","Data":"174ffb4d29f3e76f22015ffc02405c10a6381065bcc6b73ed70edf460e994c57"} Jul 08 19:20:38.488984 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:38.488931 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" podStartSLOduration=2.488915617 podStartE2EDuration="2.488915617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:20:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:20:38.486752955 +0000 UTC m=+2099.334027636" watchObservedRunningTime="2026-07-08 19:20:38.488915617 +0000 UTC m=+2099.336190263" Jul 08 19:20:39.286664 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:39.286625 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:20:39.291697 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:39.291671 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:20:39.467407 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:39.467380 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:20:39.468403 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:39.468380 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:20:46.385430 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:46.385394 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:20:54.468890 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:54.468850 2570 patch_prober.go:28] interesting pod/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Jul 08 19:20:54.469344 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:54.468910 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" podUID="35c8b38e-f351-4a13-ba10-5cfbdd7f0acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Jul 08 19:20:54.557999 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:54.557963 2570 generic.go:356] "Generic (PLEG): container finished" podID="35c8b38e-f351-4a13-ba10-5cfbdd7f0acd" containerID="174ffb4d29f3e76f22015ffc02405c10a6381065bcc6b73ed70edf460e994c57" exitCode=0 Jul 08 19:20:54.558184 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:54.558041 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" event={"ID":"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd","Type":"ContainerDied","Data":"174ffb4d29f3e76f22015ffc02405c10a6381065bcc6b73ed70edf460e994c57"} Jul 08 19:20:54.558450 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:54.558436 2570 scope.go:117] "RemoveContainer" containerID="174ffb4d29f3e76f22015ffc02405c10a6381065bcc6b73ed70edf460e994c57" Jul 08 19:20:54.559448 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:54.559419 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:20:54.559548 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:54.559489 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:20:54.559548 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:54.559523 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:20:55.566590 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:55.566555 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" event={"ID":"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd","Type":"ContainerStarted","Data":"cd76be34553e212efc498c6ac89a19c1f8b5d85447f649b31079cee71e6378eb"} Jul 08 19:20:56.286609 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:56.286574 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:20:56.286804 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:56.286780 2570 patch_prober.go:28] interesting pod/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Jul 08 19:20:56.286853 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:56.286831 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" podUID="35c8b38e-f351-4a13-ba10-5cfbdd7f0acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Jul 08 19:20:57.287303 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:57.287249 2570 patch_prober.go:28] interesting pod/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Jul 08 19:20:57.287717 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:57.287325 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" podUID="35c8b38e-f351-4a13-ba10-5cfbdd7f0acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Jul 08 19:20:58.286991 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:58.286952 2570 patch_prober.go:28] interesting pod/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Jul 08 19:20:58.287185 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:58.287010 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" podUID="35c8b38e-f351-4a13-ba10-5cfbdd7f0acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Jul 08 19:20:59.286695 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:59.286653 2570 patch_prober.go:28] interesting pod/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Jul 08 19:20:59.287109 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:20:59.286715 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" podUID="35c8b38e-f351-4a13-ba10-5cfbdd7f0acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Jul 08 19:21:00.291561 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:21:00.291526 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:21:00.292031 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:21:00.291973 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:21:00.292765 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:21:00.292747 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:23:54.204735 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:23:54.204699 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:24:03.718530 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:03.718500 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb"] Jul 08 19:24:03.718945 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:03.718708 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" podUID="35c8b38e-f351-4a13-ba10-5cfbdd7f0acd" containerName="istio-proxy" containerID="cri-o://cd76be34553e212efc498c6ac89a19c1f8b5d85447f649b31079cee71e6378eb" gracePeriod=30 Jul 08 19:24:05.795720 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:05.795673 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-rg45r"] Jul 08 19:24:08.992513 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:08.992451 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:24:08.995670 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:08.995650 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-istio-data\") pod \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\" (UID: \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\") " Jul 08 19:24:08.995729 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:08.995698 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-workload-socket\") pod \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\" (UID: \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\") " Jul 08 19:24:08.995765 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:08.995743 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-istio-token\") pod \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\" (UID: \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\") " Jul 08 19:24:08.995810 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:08.995769 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-istio-podinfo\") pod \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\" (UID: \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\") " Jul 08 19:24:08.995810 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:08.995793 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-istio-envoy\") pod \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\" (UID: \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\") " Jul 08 19:24:08.995910 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:08.995809 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhmh7\" (UniqueName: \"kubernetes.io/projected/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-kube-api-access-xhmh7\") pod \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\" (UID: \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\") " Jul 08 19:24:08.995910 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:08.995846 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-istiod-ca-cert\") pod \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\" (UID: \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\") " Jul 08 19:24:08.996014 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:08.995908 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-workload-certs\") pod \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\" (UID: \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\") " Jul 08 19:24:08.996014 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:08.995949 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-credential-socket\") pod \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\" (UID: \"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd\") " Jul 08 19:24:08.996014 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:08.995951 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "35c8b38e-f351-4a13-ba10-5cfbdd7f0acd" (UID: "35c8b38e-f351-4a13-ba10-5cfbdd7f0acd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:24:08.996197 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:08.996179 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "35c8b38e-f351-4a13-ba10-5cfbdd7f0acd" (UID: "35c8b38e-f351-4a13-ba10-5cfbdd7f0acd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:24:08.996568 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:08.996317 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-workload-socket\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:24:08.996568 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:08.996340 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-workload-certs\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:24:08.996568 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:08.996512 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "35c8b38e-f351-4a13-ba10-5cfbdd7f0acd" (UID: "35c8b38e-f351-4a13-ba10-5cfbdd7f0acd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:24:08.996568 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:08.996525 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-istio-data" (OuterVolumeSpecName: "istio-data") pod "35c8b38e-f351-4a13-ba10-5cfbdd7f0acd" (UID: "35c8b38e-f351-4a13-ba10-5cfbdd7f0acd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:24:08.996568 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:08.996537 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "35c8b38e-f351-4a13-ba10-5cfbdd7f0acd" (UID: "35c8b38e-f351-4a13-ba10-5cfbdd7f0acd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:24:08.998181 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:08.998151 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-kube-api-access-xhmh7" (OuterVolumeSpecName: "kube-api-access-xhmh7") pod "35c8b38e-f351-4a13-ba10-5cfbdd7f0acd" (UID: "35c8b38e-f351-4a13-ba10-5cfbdd7f0acd"). InnerVolumeSpecName "kube-api-access-xhmh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:24:08.998327 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:08.998225 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "35c8b38e-f351-4a13-ba10-5cfbdd7f0acd" (UID: "35c8b38e-f351-4a13-ba10-5cfbdd7f0acd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:24:08.998481 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:08.998450 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "35c8b38e-f351-4a13-ba10-5cfbdd7f0acd" (UID: "35c8b38e-f351-4a13-ba10-5cfbdd7f0acd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 19:24:08.998481 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:08.998458 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-istio-token" (OuterVolumeSpecName: "istio-token") pod "35c8b38e-f351-4a13-ba10-5cfbdd7f0acd" (UID: "35c8b38e-f351-4a13-ba10-5cfbdd7f0acd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:24:09.097129 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:09.097089 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-istio-token\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:24:09.097129 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:09.097121 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-istio-podinfo\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:24:09.097129 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:09.097134 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-istio-envoy\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:24:09.097421 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:09.097147 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhmh7\" (UniqueName: \"kubernetes.io/projected/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-kube-api-access-xhmh7\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:24:09.097421 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:09.097161 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-istiod-ca-cert\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:24:09.097421 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:09.097173 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-credential-socket\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:24:09.097421 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:09.097184 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd-istio-data\") on node \"ip-10-0-130-48.ec2.internal\" DevicePath \"\"" Jul 08 19:24:09.444147 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:09.444117 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-cffw4_b9ebce16-901d-48e0-90fd-42cb587f20fc/manager/0.log" Jul 08 19:24:09.570392 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:09.570358 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-trp94_0b7b3f4d-3e6f-4045-a0cb-041d055fe403/manager/0.log" Jul 08 19:24:09.673931 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:09.673894 2570 generic.go:356] "Generic (PLEG): container finished" podID="35c8b38e-f351-4a13-ba10-5cfbdd7f0acd" containerID="cd76be34553e212efc498c6ac89a19c1f8b5d85447f649b31079cee71e6378eb" exitCode=0 Jul 08 19:24:09.674124 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:09.673952 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" Jul 08 19:24:09.674124 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:09.673960 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" event={"ID":"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd","Type":"ContainerDied","Data":"cd76be34553e212efc498c6ac89a19c1f8b5d85447f649b31079cee71e6378eb"} Jul 08 19:24:09.674124 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:09.674016 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb" event={"ID":"35c8b38e-f351-4a13-ba10-5cfbdd7f0acd","Type":"ContainerDied","Data":"51caba670fc4d5f48222b87814ff68e4609c6d17542eee05773c05f6a1671e3a"} Jul 08 19:24:09.674124 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:09.674041 2570 scope.go:117] "RemoveContainer" containerID="cd76be34553e212efc498c6ac89a19c1f8b5d85447f649b31079cee71e6378eb" Jul 08 19:24:09.684158 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:09.684133 2570 scope.go:117] "RemoveContainer" containerID="174ffb4d29f3e76f22015ffc02405c10a6381065bcc6b73ed70edf460e994c57" Jul 08 19:24:09.695651 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:09.695623 2570 scope.go:117] "RemoveContainer" containerID="cd76be34553e212efc498c6ac89a19c1f8b5d85447f649b31079cee71e6378eb" Jul 08 19:24:09.696439 ip-10-0-130-48 kubenswrapper[2570]: E0708 19:24:09.696403 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd76be34553e212efc498c6ac89a19c1f8b5d85447f649b31079cee71e6378eb\": container with ID starting with cd76be34553e212efc498c6ac89a19c1f8b5d85447f649b31079cee71e6378eb not found: ID does not exist" containerID="cd76be34553e212efc498c6ac89a19c1f8b5d85447f649b31079cee71e6378eb" Jul 08 19:24:09.696680 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:09.696447 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd76be34553e212efc498c6ac89a19c1f8b5d85447f649b31079cee71e6378eb"} err="failed to get container status \"cd76be34553e212efc498c6ac89a19c1f8b5d85447f649b31079cee71e6378eb\": rpc error: code = NotFound desc = could not find container \"cd76be34553e212efc498c6ac89a19c1f8b5d85447f649b31079cee71e6378eb\": container with ID starting with cd76be34553e212efc498c6ac89a19c1f8b5d85447f649b31079cee71e6378eb not found: ID does not exist" Jul 08 19:24:09.696680 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:09.696472 2570 scope.go:117] "RemoveContainer" containerID="174ffb4d29f3e76f22015ffc02405c10a6381065bcc6b73ed70edf460e994c57" Jul 08 19:24:09.696853 ip-10-0-130-48 kubenswrapper[2570]: E0708 19:24:09.696754 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"174ffb4d29f3e76f22015ffc02405c10a6381065bcc6b73ed70edf460e994c57\": container with ID starting with 174ffb4d29f3e76f22015ffc02405c10a6381065bcc6b73ed70edf460e994c57 not found: ID does not exist" containerID="174ffb4d29f3e76f22015ffc02405c10a6381065bcc6b73ed70edf460e994c57" Jul 08 19:24:09.696853 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:09.696793 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"174ffb4d29f3e76f22015ffc02405c10a6381065bcc6b73ed70edf460e994c57"} err="failed to get container status \"174ffb4d29f3e76f22015ffc02405c10a6381065bcc6b73ed70edf460e994c57\": rpc error: code = NotFound desc = could not find container \"174ffb4d29f3e76f22015ffc02405c10a6381065bcc6b73ed70edf460e994c57\": container with ID starting with 174ffb4d29f3e76f22015ffc02405c10a6381065bcc6b73ed70edf460e994c57 not found: ID does not exist" Jul 08 19:24:09.702617 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:09.702594 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb"] Jul 08 19:24:09.702723 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:09.702694 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-b28b28-openshift-default-86668d66ff-rvqvb"] Jul 08 19:24:09.705517 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:09.705498 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6db59b6dbc-q7l65_92caa313-e898-4633-b6f6-fb6b24aae095/manager/0.log" Jul 08 19:24:09.948111 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:09.948029 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-9xllh_df7227ff-4ae1-4f9c-8903-860139d8a70e/manager/0.log" Jul 08 19:24:10.069937 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:10.069904 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bb8746f75-2qk9m_9d608d49-4650-4c31-a350-bbcbe95baef2/copy-manifests/0.log" Jul 08 19:24:10.082560 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:10.082534 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bb8746f75-2qk9m_9d608d49-4650-4c31-a350-bbcbe95baef2/manager/0.log" Jul 08 19:24:11.172941 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:11.172905 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h27wj_acf7738b-2bc0-41f9-862a-30e9b72ae410/util/0.log" Jul 08 19:24:11.187022 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:11.187003 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h27wj_acf7738b-2bc0-41f9-862a-30e9b72ae410/pull/0.log" Jul 08 19:24:11.200390 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:11.200360 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h27wj_acf7738b-2bc0-41f9-862a-30e9b72ae410/extract/0.log" Jul 08 19:24:11.316701 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:11.316666 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfzkd_666ebeb8-9b4e-4ec6-8f30-f27e760bf720/util/0.log" Jul 08 19:24:11.322999 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:11.322976 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfzkd_666ebeb8-9b4e-4ec6-8f30-f27e760bf720/pull/0.log" Jul 08 19:24:11.329526 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:11.329508 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfzkd_666ebeb8-9b4e-4ec6-8f30-f27e760bf720/extract/0.log" Jul 08 19:24:11.448511 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:11.448423 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vbp48_55c0a620-41c0-4f36-ac2d-8a7f649b0216/util/0.log" Jul 08 19:24:11.454373 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:11.454350 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vbp48_55c0a620-41c0-4f36-ac2d-8a7f649b0216/pull/0.log" Jul 08 19:24:11.460332 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:11.460312 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vbp48_55c0a620-41c0-4f36-ac2d-8a7f649b0216/extract/0.log" Jul 08 19:24:11.571502 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:11.571468 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bq554_3c2e7ae4-1be9-450b-9d0b-52d16d2108b8/extract/0.log" Jul 08 19:24:11.577009 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:11.576988 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bq554_3c2e7ae4-1be9-450b-9d0b-52d16d2108b8/util/0.log" Jul 08 19:24:11.583181 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:11.583160 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bq554_3c2e7ae4-1be9-450b-9d0b-52d16d2108b8/pull/0.log" Jul 08 19:24:11.689868 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:11.689833 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35c8b38e-f351-4a13-ba10-5cfbdd7f0acd" path="/var/lib/kubelet/pods/35c8b38e-f351-4a13-ba10-5cfbdd7f0acd/volumes" Jul 08 19:24:11.830376 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:11.830284 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-qsplg_4f2ed85e-3619-441c-ac5d-f1608dcec2c3/manager/0.log" Jul 08 19:24:11.952220 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:11.952192 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-br2mx_1ea34d8d-e04b-4fad-b8f7-f48449f8e1d3/manager/0.log" Jul 08 19:24:12.070776 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:12.070749 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-nxbdl_e076259a-01e3-4020-bae5-6959ef897279/kuadrant-console-plugin/0.log" Jul 08 19:24:12.190431 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:12.190403 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-2q2kh_7c02bba5-df73-4c69-ac59-269c874ecf10/registry-server/0.log" Jul 08 19:24:12.316851 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:12.316819 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-dh7sh_4183c419-1826-4851-9952-d26a07ed80aa/manager/0.log" Jul 08 19:24:12.443193 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:12.443096 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-rg45r_9bf4e223-1f2a-4b18-82c4-47a92a81eb70/limitador/0.log" Jul 08 19:24:13.299129 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:13.299099 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64798db975-sccdk_85943ce7-bf4a-45e9-9342-36d6aae74b29/kube-auth-proxy/0.log" Jul 08 19:24:13.424408 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:13.424377 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-g96h8_d8a2a709-efb7-4820-831e-178d5bfb30c8/istio-proxy/1.log" Jul 08 19:24:13.673515 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:13.673486 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-gj4nt_03edc411-3219-4b5a-8b2b-16d7b9b593f4/payload-processing/0.log" Jul 08 19:24:19.236514 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:19.236476 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-87fkv/must-gather-9jzsv"] Jul 08 19:24:19.236929 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:19.236911 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35c8b38e-f351-4a13-ba10-5cfbdd7f0acd" containerName="istio-proxy" Jul 08 19:24:19.236929 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:19.236924 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="35c8b38e-f351-4a13-ba10-5cfbdd7f0acd" containerName="istio-proxy" Jul 08 19:24:19.237021 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:19.236941 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35c8b38e-f351-4a13-ba10-5cfbdd7f0acd" containerName="istio-proxy" Jul 08 19:24:19.237021 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:19.236946 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="35c8b38e-f351-4a13-ba10-5cfbdd7f0acd" containerName="istio-proxy" Jul 08 19:24:19.237021 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:19.237020 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="35c8b38e-f351-4a13-ba10-5cfbdd7f0acd" containerName="istio-proxy" Jul 08 19:24:19.237125 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:19.237029 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="35c8b38e-f351-4a13-ba10-5cfbdd7f0acd" containerName="istio-proxy" Jul 08 19:24:19.237125 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:19.237037 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="1acd12b3-a32e-4827-bdca-f6918ec56586" containerName="istio-proxy" Jul 08 19:24:19.240524 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:19.240499 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-87fkv/must-gather-9jzsv" Jul 08 19:24:19.243055 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:19.243032 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-87fkv\"/\"openshift-service-ca.crt\"" Jul 08 19:24:19.243185 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:19.243127 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-87fkv\"/\"default-dockercfg-s4hc9\"" Jul 08 19:24:19.244002 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:19.243984 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-87fkv\"/\"kube-root-ca.crt\"" Jul 08 19:24:19.248191 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:19.248147 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-87fkv/must-gather-9jzsv"] Jul 08 19:24:19.292046 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:19.292012 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxh2g\" (UniqueName: \"kubernetes.io/projected/2a27a56c-96ab-4bf2-9087-f0ae51f2c7e8-kube-api-access-sxh2g\") pod \"must-gather-9jzsv\" (UID: \"2a27a56c-96ab-4bf2-9087-f0ae51f2c7e8\") " pod="openshift-must-gather-87fkv/must-gather-9jzsv" Jul 08 19:24:19.292224 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:19.292130 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2a27a56c-96ab-4bf2-9087-f0ae51f2c7e8-must-gather-output\") pod \"must-gather-9jzsv\" (UID: \"2a27a56c-96ab-4bf2-9087-f0ae51f2c7e8\") " pod="openshift-must-gather-87fkv/must-gather-9jzsv" Jul 08 19:24:19.393592 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:19.393553 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2a27a56c-96ab-4bf2-9087-f0ae51f2c7e8-must-gather-output\") pod \"must-gather-9jzsv\" (UID: \"2a27a56c-96ab-4bf2-9087-f0ae51f2c7e8\") " pod="openshift-must-gather-87fkv/must-gather-9jzsv" Jul 08 19:24:19.393772 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:19.393621 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxh2g\" (UniqueName: \"kubernetes.io/projected/2a27a56c-96ab-4bf2-9087-f0ae51f2c7e8-kube-api-access-sxh2g\") pod \"must-gather-9jzsv\" (UID: \"2a27a56c-96ab-4bf2-9087-f0ae51f2c7e8\") " pod="openshift-must-gather-87fkv/must-gather-9jzsv" Jul 08 19:24:19.393892 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:19.393872 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2a27a56c-96ab-4bf2-9087-f0ae51f2c7e8-must-gather-output\") pod \"must-gather-9jzsv\" (UID: \"2a27a56c-96ab-4bf2-9087-f0ae51f2c7e8\") " pod="openshift-must-gather-87fkv/must-gather-9jzsv" Jul 08 19:24:19.402657 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:19.402629 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxh2g\" (UniqueName: \"kubernetes.io/projected/2a27a56c-96ab-4bf2-9087-f0ae51f2c7e8-kube-api-access-sxh2g\") pod \"must-gather-9jzsv\" (UID: \"2a27a56c-96ab-4bf2-9087-f0ae51f2c7e8\") " pod="openshift-must-gather-87fkv/must-gather-9jzsv" Jul 08 19:24:19.550955 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:19.550866 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-87fkv/must-gather-9jzsv" Jul 08 19:24:19.691313 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:19.691288 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-87fkv/must-gather-9jzsv"] Jul 08 19:24:19.693029 ip-10-0-130-48 kubenswrapper[2570]: W0708 19:24:19.693000 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a27a56c_96ab_4bf2_9087_f0ae51f2c7e8.slice/crio-2cd61bca48b2f719564df63d6b429188a7aa277411c657e96fd94070a1155550 WatchSource:0}: Error finding container 2cd61bca48b2f719564df63d6b429188a7aa277411c657e96fd94070a1155550: Status 404 returned error can't find the container with id 2cd61bca48b2f719564df63d6b429188a7aa277411c657e96fd94070a1155550 Jul 08 19:24:19.694894 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:19.694876 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 19:24:19.732000 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:19.731959 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-87fkv/must-gather-9jzsv" event={"ID":"2a27a56c-96ab-4bf2-9087-f0ae51f2c7e8","Type":"ContainerStarted","Data":"2cd61bca48b2f719564df63d6b429188a7aa277411c657e96fd94070a1155550"} Jul 08 19:24:21.748348 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:21.748305 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-87fkv/must-gather-9jzsv" event={"ID":"2a27a56c-96ab-4bf2-9087-f0ae51f2c7e8","Type":"ContainerStarted","Data":"078ed974da12903410fb057c50b3c72c5568b6d35783d3264de74bacfb6b04fa"} Jul 08 19:24:21.748348 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:21.748363 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-87fkv/must-gather-9jzsv" event={"ID":"2a27a56c-96ab-4bf2-9087-f0ae51f2c7e8","Type":"ContainerStarted","Data":"637fce8132bbd0fb7a0a3558712e8431ff39af151be40e1616409142281c77f8"} Jul 08 19:24:21.767604 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:21.767538 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-87fkv/must-gather-9jzsv" podStartSLOduration=1.797092985 podStartE2EDuration="2.767515764s" totalImagesPullingTime="970.422779ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:24:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:24:21.765151927 +0000 UTC m=+2322.612426594" watchObservedRunningTime="2026-07-08 19:24:21.767515764 +0000 UTC m=+2322.614790432" Jul 08 19:24:22.226732 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:22.226686 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9gjk9_02e604f5-7b5c-4aec-a685-e380d4b20a3e/global-pull-secret-syncer/0.log" Jul 08 19:24:22.387879 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:22.387844 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tfrw6_d8fdda41-866c-40a9-95ff-599de68e4672/konnectivity-agent/0.log" Jul 08 19:24:22.408974 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:22.408946 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-48.ec2.internal_258c89e0ffadf0a34366b9c773cb3f43/haproxy/0.log" Jul 08 19:24:27.080065 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:27.080022 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h27wj_acf7738b-2bc0-41f9-862a-30e9b72ae410/extract/0.log" Jul 08 19:24:27.107129 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:27.107099 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h27wj_acf7738b-2bc0-41f9-862a-30e9b72ae410/util/0.log" Jul 08 19:24:27.129472 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:27.129445 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759h27wj_acf7738b-2bc0-41f9-862a-30e9b72ae410/pull/0.log" Jul 08 19:24:27.154169 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:27.154088 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfzkd_666ebeb8-9b4e-4ec6-8f30-f27e760bf720/extract/0.log" Jul 08 19:24:27.175501 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:27.175463 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfzkd_666ebeb8-9b4e-4ec6-8f30-f27e760bf720/util/0.log" Jul 08 19:24:27.203321 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:27.203248 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfzkd_666ebeb8-9b4e-4ec6-8f30-f27e760bf720/pull/0.log" Jul 08 19:24:27.227080 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:27.227016 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vbp48_55c0a620-41c0-4f36-ac2d-8a7f649b0216/extract/0.log" Jul 08 19:24:27.248809 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:27.248785 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vbp48_55c0a620-41c0-4f36-ac2d-8a7f649b0216/util/0.log" Jul 08 19:24:27.276233 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:27.276185 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vbp48_55c0a620-41c0-4f36-ac2d-8a7f649b0216/pull/0.log" Jul 08 19:24:27.330339 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:27.330237 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bq554_3c2e7ae4-1be9-450b-9d0b-52d16d2108b8/extract/0.log" Jul 08 19:24:27.354287 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:27.354237 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bq554_3c2e7ae4-1be9-450b-9d0b-52d16d2108b8/util/0.log" Jul 08 19:24:27.376817 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:27.376782 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bq554_3c2e7ae4-1be9-450b-9d0b-52d16d2108b8/pull/0.log" Jul 08 19:24:27.676720 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:27.676670 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-qsplg_4f2ed85e-3619-441c-ac5d-f1608dcec2c3/manager/0.log" Jul 08 19:24:27.705854 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:27.705821 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-br2mx_1ea34d8d-e04b-4fad-b8f7-f48449f8e1d3/manager/0.log" Jul 08 19:24:27.736873 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:27.736835 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-nxbdl_e076259a-01e3-4020-bae5-6959ef897279/kuadrant-console-plugin/0.log" Jul 08 19:24:27.783277 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:27.783010 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-2q2kh_7c02bba5-df73-4c69-ac59-269c874ecf10/registry-server/0.log" Jul 08 19:24:27.944434 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:27.943317 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-dh7sh_4183c419-1826-4851-9952-d26a07ed80aa/manager/0.log" Jul 08 19:24:27.991286 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:27.989924 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-rg45r_9bf4e223-1f2a-4b18-82c4-47a92a81eb70/limitador/0.log" Jul 08 19:24:30.039858 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:30.039789 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6bcb9bcc4b-5wrlm_9b81ec53-6553-4e90-9013-57f222fad5e6/metrics-server/0.log" Jul 08 19:24:30.074836 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:30.074809 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-lk2jp_e0b7951b-9599-4e80-bf70-173c5a4cf310/monitoring-plugin/0.log" Jul 08 19:24:30.194447 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:30.194373 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ghhs2_14933c7c-f7ba-40d9-8ede-f7bd241a9a38/node-exporter/0.log" Jul 08 19:24:30.220958 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:30.220918 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ghhs2_14933c7c-f7ba-40d9-8ede-f7bd241a9a38/kube-rbac-proxy/0.log" Jul 08 19:24:30.244143 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:30.244110 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ghhs2_14933c7c-f7ba-40d9-8ede-f7bd241a9a38/init-textfile/0.log" Jul 08 19:24:30.612281 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:30.611808 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-n588z_ca8d865e-4b64-4bda-8030-19555f2cba03/prometheus-operator/0.log" Jul 08 19:24:30.633806 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:30.633771 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-n588z_ca8d865e-4b64-4bda-8030-19555f2cba03/kube-rbac-proxy/0.log" Jul 08 19:24:30.661576 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:30.661535 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-kxhqr_fccc84da-2d6a-4559-bffb-d0f5958e8471/prometheus-operator-admission-webhook/0.log" Jul 08 19:24:30.791290 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:30.791244 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7756484b49-bg58n_a41d4b66-d1a1-4393-83a8-de9bbe36c08a/thanos-query/0.log" Jul 08 19:24:30.810897 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:30.810869 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7756484b49-bg58n_a41d4b66-d1a1-4393-83a8-de9bbe36c08a/kube-rbac-proxy-web/0.log" Jul 08 19:24:30.831706 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:30.831674 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7756484b49-bg58n_a41d4b66-d1a1-4393-83a8-de9bbe36c08a/kube-rbac-proxy/0.log" Jul 08 19:24:30.851864 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:30.851839 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7756484b49-bg58n_a41d4b66-d1a1-4393-83a8-de9bbe36c08a/prom-label-proxy/0.log" Jul 08 19:24:30.872644 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:30.872546 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7756484b49-bg58n_a41d4b66-d1a1-4393-83a8-de9bbe36c08a/kube-rbac-proxy-rules/0.log" Jul 08 19:24:30.895561 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:30.895528 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7756484b49-bg58n_a41d4b66-d1a1-4393-83a8-de9bbe36c08a/kube-rbac-proxy-metrics/0.log" Jul 08 19:24:32.060583 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:32.060556 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-bbrn2_ed038c0a-ac37-4723-abee-8b820d63e4bd/networking-console-plugin/0.log" Jul 08 19:24:32.923035 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:32.922999 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-87fkv/perf-node-gather-daemonset-gqq67"] Jul 08 19:24:32.940652 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:32.940613 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-87fkv/perf-node-gather-daemonset-gqq67"] Jul 08 19:24:32.940874 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:32.940776 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-87fkv/perf-node-gather-daemonset-gqq67" Jul 08 19:24:33.045625 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:33.045561 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/565b5cba-1d2b-49d5-8011-5b2dd6573c2c-proc\") pod \"perf-node-gather-daemonset-gqq67\" (UID: \"565b5cba-1d2b-49d5-8011-5b2dd6573c2c\") " pod="openshift-must-gather-87fkv/perf-node-gather-daemonset-gqq67" Jul 08 19:24:33.045829 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:33.045630 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/565b5cba-1d2b-49d5-8011-5b2dd6573c2c-lib-modules\") pod \"perf-node-gather-daemonset-gqq67\" (UID: \"565b5cba-1d2b-49d5-8011-5b2dd6573c2c\") " pod="openshift-must-gather-87fkv/perf-node-gather-daemonset-gqq67" Jul 08 19:24:33.045829 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:33.045677 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhcrv\" (UniqueName: \"kubernetes.io/projected/565b5cba-1d2b-49d5-8011-5b2dd6573c2c-kube-api-access-hhcrv\") pod \"perf-node-gather-daemonset-gqq67\" (UID: \"565b5cba-1d2b-49d5-8011-5b2dd6573c2c\") " pod="openshift-must-gather-87fkv/perf-node-gather-daemonset-gqq67" Jul 08 19:24:33.045829 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:33.045777 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/565b5cba-1d2b-49d5-8011-5b2dd6573c2c-podres\") pod \"perf-node-gather-daemonset-gqq67\" (UID: \"565b5cba-1d2b-49d5-8011-5b2dd6573c2c\") " pod="openshift-must-gather-87fkv/perf-node-gather-daemonset-gqq67" Jul 08 19:24:33.046010 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:33.045849 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/565b5cba-1d2b-49d5-8011-5b2dd6573c2c-sys\") pod \"perf-node-gather-daemonset-gqq67\" (UID: \"565b5cba-1d2b-49d5-8011-5b2dd6573c2c\") " pod="openshift-must-gather-87fkv/perf-node-gather-daemonset-gqq67" Jul 08 19:24:33.146674 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:33.146625 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/565b5cba-1d2b-49d5-8011-5b2dd6573c2c-lib-modules\") pod \"perf-node-gather-daemonset-gqq67\" (UID: \"565b5cba-1d2b-49d5-8011-5b2dd6573c2c\") " pod="openshift-must-gather-87fkv/perf-node-gather-daemonset-gqq67" Jul 08 19:24:33.147158 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:33.146684 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhcrv\" (UniqueName: \"kubernetes.io/projected/565b5cba-1d2b-49d5-8011-5b2dd6573c2c-kube-api-access-hhcrv\") pod \"perf-node-gather-daemonset-gqq67\" (UID: \"565b5cba-1d2b-49d5-8011-5b2dd6573c2c\") " pod="openshift-must-gather-87fkv/perf-node-gather-daemonset-gqq67" Jul 08 19:24:33.147158 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:33.146749 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/565b5cba-1d2b-49d5-8011-5b2dd6573c2c-podres\") pod \"perf-node-gather-daemonset-gqq67\" (UID: \"565b5cba-1d2b-49d5-8011-5b2dd6573c2c\") " pod="openshift-must-gather-87fkv/perf-node-gather-daemonset-gqq67" Jul 08 19:24:33.147158 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:33.146808 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/565b5cba-1d2b-49d5-8011-5b2dd6573c2c-sys\") pod \"perf-node-gather-daemonset-gqq67\" (UID: \"565b5cba-1d2b-49d5-8011-5b2dd6573c2c\") " pod="openshift-must-gather-87fkv/perf-node-gather-daemonset-gqq67" Jul 08 19:24:33.147158 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:33.146910 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/565b5cba-1d2b-49d5-8011-5b2dd6573c2c-proc\") pod \"perf-node-gather-daemonset-gqq67\" (UID: \"565b5cba-1d2b-49d5-8011-5b2dd6573c2c\") " pod="openshift-must-gather-87fkv/perf-node-gather-daemonset-gqq67" Jul 08 19:24:33.147158 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:33.147014 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/565b5cba-1d2b-49d5-8011-5b2dd6573c2c-proc\") pod \"perf-node-gather-daemonset-gqq67\" (UID: \"565b5cba-1d2b-49d5-8011-5b2dd6573c2c\") " pod="openshift-must-gather-87fkv/perf-node-gather-daemonset-gqq67" Jul 08 19:24:33.147158 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:33.147139 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/565b5cba-1d2b-49d5-8011-5b2dd6573c2c-lib-modules\") pod \"perf-node-gather-daemonset-gqq67\" (UID: \"565b5cba-1d2b-49d5-8011-5b2dd6573c2c\") " pod="openshift-must-gather-87fkv/perf-node-gather-daemonset-gqq67" Jul 08 19:24:33.147600 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:33.147575 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/565b5cba-1d2b-49d5-8011-5b2dd6573c2c-podres\") pod \"perf-node-gather-daemonset-gqq67\" (UID: \"565b5cba-1d2b-49d5-8011-5b2dd6573c2c\") " pod="openshift-must-gather-87fkv/perf-node-gather-daemonset-gqq67" Jul 08 19:24:33.147663 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:33.147646 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/565b5cba-1d2b-49d5-8011-5b2dd6573c2c-sys\") pod \"perf-node-gather-daemonset-gqq67\" (UID: \"565b5cba-1d2b-49d5-8011-5b2dd6573c2c\") " pod="openshift-must-gather-87fkv/perf-node-gather-daemonset-gqq67" Jul 08 19:24:33.158747 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:33.158712 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhcrv\" (UniqueName: \"kubernetes.io/projected/565b5cba-1d2b-49d5-8011-5b2dd6573c2c-kube-api-access-hhcrv\") pod \"perf-node-gather-daemonset-gqq67\" (UID: \"565b5cba-1d2b-49d5-8011-5b2dd6573c2c\") " pod="openshift-must-gather-87fkv/perf-node-gather-daemonset-gqq67" Jul 08 19:24:33.244760 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:33.244666 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-68d45ff57c-7m45k_46ebf7d1-1bc3-43de-bfd1-a4d7d03babfa/console/0.log" Jul 08 19:24:33.264528 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:33.264478 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-87fkv/perf-node-gather-daemonset-gqq67" Jul 08 19:24:33.278180 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:33.278143 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-fx4jm_d11e5bed-ffc0-483d-ad13-435bc9131d71/download-server/0.log" Jul 08 19:24:33.481154 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:33.481078 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-87fkv/perf-node-gather-daemonset-gqq67"] Jul 08 19:24:33.485074 ip-10-0-130-48 kubenswrapper[2570]: W0708 19:24:33.485040 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod565b5cba_1d2b_49d5_8011_5b2dd6573c2c.slice/crio-688914a1f2fc69421c07f70a7d447fbb446390997233a9af2c6caf9d0ea77aed WatchSource:0}: Error finding container 688914a1f2fc69421c07f70a7d447fbb446390997233a9af2c6caf9d0ea77aed: Status 404 returned error can't find the container with id 688914a1f2fc69421c07f70a7d447fbb446390997233a9af2c6caf9d0ea77aed Jul 08 19:24:33.862376 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:33.862338 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-87fkv/perf-node-gather-daemonset-gqq67" event={"ID":"565b5cba-1d2b-49d5-8011-5b2dd6573c2c","Type":"ContainerStarted","Data":"49891537f1efa8fc7929d39db249a467cc691d0b35fe9c8bb8d0455234af87d9"} Jul 08 19:24:33.862572 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:33.862414 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-87fkv/perf-node-gather-daemonset-gqq67" event={"ID":"565b5cba-1d2b-49d5-8011-5b2dd6573c2c","Type":"ContainerStarted","Data":"688914a1f2fc69421c07f70a7d447fbb446390997233a9af2c6caf9d0ea77aed"} Jul 08 19:24:33.862572 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:33.862534 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-87fkv/perf-node-gather-daemonset-gqq67" Jul 08 19:24:35.110521 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:35.110494 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rhwrh_5f0490c6-dc1a-4e6e-8e53-e9a0eb220119/dns/0.log" Jul 08 19:24:35.131147 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:35.131116 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rhwrh_5f0490c6-dc1a-4e6e-8e53-e9a0eb220119/kube-rbac-proxy/0.log" Jul 08 19:24:35.198038 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:35.198009 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8hxcn_f79dc66e-2b96-42c4-9ee9-4d036cc63d3a/dns-node-resolver/0.log" Jul 08 19:24:35.799732 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:35.799705 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-984989766-ktxtr_ce0fcade-7b07-4a9a-b843-9937f2de7038/registry/0.log" Jul 08 19:24:35.865785 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:35.865758 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dqz5x_7a28d7e6-edce-4adf-8200-d499f2b365b0/node-ca/0.log" Jul 08 19:24:37.002955 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:37.002924 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64798db975-sccdk_85943ce7-bf4a-45e9-9342-36d6aae74b29/kube-auth-proxy/0.log" Jul 08 19:24:37.029097 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:37.029071 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-g96h8_d8a2a709-efb7-4820-831e-178d5bfb30c8/istio-proxy/0.log" Jul 08 19:24:37.067109 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:37.067081 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-g96h8_d8a2a709-efb7-4820-831e-178d5bfb30c8/istio-proxy/1.log" Jul 08 19:24:37.180044 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:37.180011 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-gj4nt_03edc411-3219-4b5a-8b2b-16d7b9b593f4/payload-processing/0.log" Jul 08 19:24:37.931321 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:37.931293 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kg6bp_bcc10be2-d0a7-47a2-9b49-0316b50cb8da/serve-healthcheck-canary/0.log" Jul 08 19:24:38.530831 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:38.530794 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-db4fh_c59f953f-8ffe-4d96-9bcc-68622e7e73f7/kube-rbac-proxy/0.log" Jul 08 19:24:38.552344 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:38.552311 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-db4fh_c59f953f-8ffe-4d96-9bcc-68622e7e73f7/exporter/0.log" Jul 08 19:24:38.578863 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:38.578814 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-db4fh_c59f953f-8ffe-4d96-9bcc-68622e7e73f7/extractor/0.log" Jul 08 19:24:39.885819 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:39.885787 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-87fkv/perf-node-gather-daemonset-gqq67" Jul 08 19:24:39.906517 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:39.906460 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-87fkv/perf-node-gather-daemonset-gqq67" podStartSLOduration=7.906439368 podStartE2EDuration="7.906439368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:24:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:24:33.891216455 +0000 UTC m=+2334.738491125" watchObservedRunningTime="2026-07-08 19:24:39.906439368 +0000 UTC m=+2340.753714035" Jul 08 19:24:40.795714 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:40.795684 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-cffw4_b9ebce16-901d-48e0-90fd-42cb587f20fc/manager/0.log" Jul 08 19:24:40.947484 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:40.947436 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-trp94_0b7b3f4d-3e6f-4045-a0cb-041d055fe403/manager/0.log" Jul 08 19:24:41.070670 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:41.070545 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6db59b6dbc-q7l65_92caa313-e898-4633-b6f6-fb6b24aae095/manager/0.log" Jul 08 19:24:41.185387 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:41.185354 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-9xllh_df7227ff-4ae1-4f9c-8903-860139d8a70e/manager/0.log" Jul 08 19:24:41.269937 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:41.269903 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bb8746f75-2qk9m_9d608d49-4650-4c31-a350-bbcbe95baef2/manager/0.log" Jul 08 19:24:41.286955 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:41.286924 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bb8746f75-2qk9m_9d608d49-4650-4c31-a350-bbcbe95baef2/copy-manifests/0.log" Jul 08 19:24:42.887665 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:42.887629 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-pk85g_78c3ef05-a6b9-4eed-89f5-8618a93bb2d8/openshift-lws-operator/0.log" Jul 08 19:24:43.191826 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:43.191786 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64798db975-sccdk_85943ce7-bf4a-45e9-9342-36d6aae74b29/kube-auth-proxy/0.log" Jul 08 19:24:43.215488 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:43.215455 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-g96h8_d8a2a709-efb7-4820-831e-178d5bfb30c8/istio-proxy/0.log" Jul 08 19:24:43.255791 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:43.255751 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-g96h8_d8a2a709-efb7-4820-831e-178d5bfb30c8/istio-proxy/1.log" Jul 08 19:24:43.357645 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:43.357565 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-gj4nt_03edc411-3219-4b5a-8b2b-16d7b9b593f4/payload-processing/0.log" Jul 08 19:24:44.682832 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:44.682797 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64798db975-sccdk_85943ce7-bf4a-45e9-9342-36d6aae74b29/kube-auth-proxy/0.log" Jul 08 19:24:44.700564 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:44.700527 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-g96h8_d8a2a709-efb7-4820-831e-178d5bfb30c8/istio-proxy/0.log" Jul 08 19:24:44.736513 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:44.736484 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-g96h8_d8a2a709-efb7-4820-831e-178d5bfb30c8/istio-proxy/1.log" Jul 08 19:24:44.838464 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:44.838377 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-gj4nt_03edc411-3219-4b5a-8b2b-16d7b9b593f4/payload-processing/0.log" Jul 08 19:24:49.582444 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:49.582416 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fdwjq_78c9e9b1-2fc2-47f5-9966-85b349a0f009/kube-multus-additional-cni-plugins/0.log" Jul 08 19:24:49.602066 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:49.602037 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fdwjq_78c9e9b1-2fc2-47f5-9966-85b349a0f009/egress-router-binary-copy/0.log" Jul 08 19:24:49.623377 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:49.623348 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fdwjq_78c9e9b1-2fc2-47f5-9966-85b349a0f009/cni-plugins/0.log" Jul 08 19:24:49.647318 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:49.647295 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fdwjq_78c9e9b1-2fc2-47f5-9966-85b349a0f009/bond-cni-plugin/0.log" Jul 08 19:24:49.667432 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:49.667407 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fdwjq_78c9e9b1-2fc2-47f5-9966-85b349a0f009/routeoverride-cni/0.log" Jul 08 19:24:49.687510 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:49.687485 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fdwjq_78c9e9b1-2fc2-47f5-9966-85b349a0f009/whereabouts-cni-bincopy/0.log" Jul 08 19:24:49.713228 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:49.713202 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fdwjq_78c9e9b1-2fc2-47f5-9966-85b349a0f009/whereabouts-cni/0.log" Jul 08 19:24:49.911512 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:49.911478 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-c2vbf_1e1cf566-6a03-4273-a870-5f78662ca68c/kube-multus/0.log" Jul 08 19:24:50.007137 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:50.007104 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mwtbr_45fe0fb7-f912-4b72-b298-1dae0ceae220/network-metrics-daemon/0.log" Jul 08 19:24:50.022880 ip-10-0-130-48 kubenswrapper[2570]: I0708 19:24:50.022854 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mwtbr_45fe0fb7-f912-4b72-b298-1dae0ceae220/kube-rbac-proxy/0.log"